-- Logs begin at Wed 2022-08-24 02:43:17 UTC, end at Wed 2022-08-24 04:09:59 UTC. -- Aug 24 03:25:18.478145 dv-cinder-854259-1-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Aug 24 03:25:18.488244 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Aug 24 03:25:18.491401 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: *** Starting uWSGI 2.0.18-debian (64bit) on [Wed Aug 24 03:25:18 2022] *** Aug 24 03:25:18.491543 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Aug 24 03:25:18.491648 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Aug 24 03:25:18.491746 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: nodename: dv-cinder-854259-1-iscsi Aug 24 03:25:18.491852 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: machine: x86_64 Aug 24 03:25:18.491852 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: clock source: unix Aug 24 03:25:18.491852 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: pcre jit disabled Aug 24 03:25:18.491852 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: detected number of CPU cores: 4 Aug 24 03:25:18.491852 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: current working directory: / Aug 24 03:25:18.491852 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: detected binary path: /usr/bin/uwsgi-core Aug 24 03:25:18.491852 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: your processes number limit is 47680 Aug 24 03:25:18.491852 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: your memory page size is 4096 bytes Aug 24 03:25:18.491852 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: detected max file descriptor number: 2048 Aug 24 03:25:18.492393 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: lock engine: pthread robust mutexes Aug 24 03:25:18.492393 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: thunder lock: enabled Aug 24 03:25:18.492393 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Aug 24 03:25:18.492393 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Aug 24 03:25:18.530736 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: Python main interpreter initialized at 0x55b7936b89c0 Aug 24 03:25:18.530736 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: python threads support enabled Aug 24 03:25:18.530736 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: your server socket listen backlog is limited to 100 connections Aug 24 03:25:18.530736 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: your mercy for graceful operations on workers is 90 seconds Aug 24 03:25:18.531086 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: mapped 403077 bytes (393 KB) for 2 cores Aug 24 03:25:18.531159 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: *** Operational MODE: preforking *** Aug 24 03:25:18.531420 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: *** uWSGI is running in multiple interpreter mode *** Aug 24 03:25:18.531420 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: spawned uWSGI master process (pid: 81617) Aug 24 03:25:18.531614 dv-cinder-854259-1-iscsi systemd[1]: Started Devstack devstack@keystone.service. Aug 24 03:25:18.535872 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: spawned uWSGI worker 1 (pid: 81657, cores: 1) Aug 24 03:25:18.535872 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: spawned uWSGI worker 2 (pid: 81658, cores: 1) Aug 24 03:25:18.535872 dv-cinder-854259-1-iscsi devstack@keystone.service[81617]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Aug 24 03:25:22.881958 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 24 03:25:22.881958 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: from cryptography.utils import int_from_bytes Aug 24 03:25:22.882755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 24 03:25:22.882755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: from cryptography.utils import int_from_bytes Aug 24 03:25:22.883213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 24 03:25:22.883213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: from cryptography.utils import int_from_bytes Aug 24 03:25:22.883671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 24 03:25:22.883671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: from cryptography.utils import int_from_bytes Aug 24 03:25:22.939353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 24 03:25:22.939538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 24 03:25:22.939622 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 24 03:25:22.939709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 24 03:25:22.939794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] command line args: [] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 24 03:25:22.939871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] command line args: [] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 24 03:25:22.939980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 24 03:25:22.940110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 24 03:25:22.940187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ================================================================================ {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 24 03:25:22.940292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ================================================================================ {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 24 03:25:22.940402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] admin_token = **** {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.940546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] admin_token = **** {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.940654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] config_dir = [] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.940792 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] config_dir = [] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.940872 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.940980 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.941056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] config_source = [] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.941161 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] config_source = [] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.941241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] debug = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.941339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] debug = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.941425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.941653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.941776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.941907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.942047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.942177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.942289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.942410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.942522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.942868 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.942868 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] list_limit = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.943209 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] list_limit = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.943358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] log_config_append = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.943358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.943551 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] log_config_append = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.943650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] log_dir = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.943727 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.943851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] log_file = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.943924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] log_dir = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.943924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] log_file = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.944103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.944216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.944288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.944411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.944484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.944596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.944675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.944675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.944859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.944987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.945061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.945061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.945239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.945354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.945435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.945504 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.945624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.945696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.945809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.945887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.946007 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.946085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.946185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.946266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.946341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.946464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.946545 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.946620 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.946740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.946898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.947151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.65/identity {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.947335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] publish_errors = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.947411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.65/identity {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.947561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] publish_errors = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.947689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.947789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.947865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.947968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.948043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.948185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.948259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.948433 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.948518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.948667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] standard_threads = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.948750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.948900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.948981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] standard_threads = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.949130 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.949203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.949306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.949392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.949482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] use_journal = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.949569 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.949640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] use_json = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.949729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] use_journal = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.949837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] use_stderr = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.949916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] use_json = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.949985 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] use_syslog = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.950057 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] use_stderr = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.950229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.953540 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] use_syslog = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.953540 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 24 03:25:22.953827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.953827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954161 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954161 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954299 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954299 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.954958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955360 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955360 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.955995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.956068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.956137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.956210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.956355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.956435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.956550 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.956622 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.956689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.956755 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.956830 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.956903 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.956971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957535 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.957958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.958050 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.958121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.958267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.958384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.958514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.958654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.958796 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.958887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.958966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.959035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.959108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.959181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.959249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.959316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.959390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.959459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.959526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.959605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.959714 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.959809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.959911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.960005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.960098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.960191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.960313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.960419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.960522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.960616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.960711 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.960826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.960906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.961035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.961157 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.961231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.961355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.961429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.961550 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.961758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.961853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.961927 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.962051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.962180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.962258 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.962374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.962448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.962574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.962644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.962713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.962833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.962936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.963056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.963179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.963257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.963420 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.963497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.963665 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.963750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.963904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.963981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.964143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.964261 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.964377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.964462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.964613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.964690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.964850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.964937 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.965085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.965166 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.965321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.965398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.965551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.965627 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.965794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.965867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.966027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.966103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.966291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.966373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.966533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.966610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.966797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.966907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.967049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.967141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.967284 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.967361 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.967526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.967605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.967765 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.967842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.968011 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.968086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.968249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.968325 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.968490 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.968572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.968729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.968806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.968966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.969058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.969206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.969282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.969443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.969520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.969680 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.969764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.969917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.969993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.970190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.970271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.970416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.970502 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.970659 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.970739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.970900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.970984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.971142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.971226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.971379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.971455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.971617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.971693 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.971854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.971938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.972085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.972270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.972348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.972418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.972536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.972622 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.972768 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.972843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.972997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.973072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.973233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.973319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.973468 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.973543 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.974256 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.974399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.974546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.974695 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.974825 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.974906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.974976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.975044 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.975200 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.975289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.975460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.975537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] auth.external = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.975706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.975782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.975942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] auth.external = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.976031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.976175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.976277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.976439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] auth.password = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.976517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.976673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] auth.token = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.976755 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.976914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.976998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] auth.password = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.977147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] auth.token = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.977223 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.977390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.977466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.977620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.977703 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.977866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.977949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.978099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.978219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.978364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.978450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.978602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] credential.caching = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.978679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.978833 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.978910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.979074 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] credential.caching = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.979160 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.979322 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.979398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.979561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.979637 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.979805 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.979893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.980041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.980118 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.980286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.980363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.980526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.980604 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.980761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.980836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.981014 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.981091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.981253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.981331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.981510 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.981587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.981780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.981855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.982024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.982104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.982308 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.982394 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.982555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] federation.caching = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.982635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.982789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.982868 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.983030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] federation.caching = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.983106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.983273 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.983350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.983505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.983582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.983754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.983831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.983995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.984191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.984339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.984439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.984519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.984589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.984722 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.984800 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.984962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.985047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.985212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.985289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.985449 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.caching = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.985525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.985696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.985774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.985935 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.caching = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.986012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.986184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.986277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.986426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.986506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.986669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.986745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.986899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.986974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.987139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.987217 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.987370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.987451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.987611 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.987689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.987846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.987923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.988078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.988155 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.988310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.988386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.988547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.988624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.988784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.988869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.989029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.989106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.989304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.989938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.990077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.990379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.990462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.990549 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.990623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.990732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.990855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.990972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.991083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.991202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.991327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.991429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.991563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.991661 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.991804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.991892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.992039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.992117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.992280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.992375 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.992523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.992609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.992765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.992843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.993016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.993126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.993250 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.993340 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.993500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.993577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.993746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.993824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.993982 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.994068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.994236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.994317 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.994477 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.994554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.994709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.995177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.995177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.995395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.995395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.995517 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.995586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.996980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.997144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.997221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.997380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.997458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.997618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.997702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.997857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.997935 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.998096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.998216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.998356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.998447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.998594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.998671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.998830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.998908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.999072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.999157 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.999322 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.999400 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.999556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.999632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.999801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:22.999885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.000037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.000115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.000270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.000346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.000505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.000593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.000735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.000811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.000969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.001046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.001201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.001289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.001448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.001526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.001696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.001774 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.001944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.002029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.002209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.002290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.002458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.002534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.002690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.002779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.002936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.003013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.003167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.003244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.003407 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.003492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.003645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.003722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.003880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.003956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.004119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.004204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.004358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.004435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.004592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.004667 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.004844 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.004955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.005136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.005279 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.005416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.005497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.005642 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.005747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.005872 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.005984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.006385 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.006510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.006631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.006762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.006924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.007001 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.007178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.007256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.007424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.007503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.007660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.007737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] resource.caching = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.007901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.007978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.008145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] resource.caching = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.008225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.008379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.008455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.008631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.008709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.008868 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.008945 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.009107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.009183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.009354 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.009430 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.009590 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.009667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.009826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.009902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.010062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] role.caching = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.010138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.010327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.010407 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] role.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.010557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.010633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] role.caching = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.010805 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] role.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.010881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.011038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.011117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.011286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.011361 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.011534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.011610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.011782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.011861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.012029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.012105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.012273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.012349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.012519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.012597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.012751 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.012827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.013000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.013125 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.013233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.013311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.013487 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.013563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.013761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.013838 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.014015 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.014174 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.014288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.014432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.014685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.014774 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.014924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.015002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.015174 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.015250 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.015423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.015508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.015662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.015740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.015910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.015986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.016148 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.016233 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.016390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.016467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.016625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.016702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.016860 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.016945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.017092 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.017171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.017324 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.017401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.017558 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.017644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.017812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.017889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.018048 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.018126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.018339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.018544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.018629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.018689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.018771 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] token.caching = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.018840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.019010 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.019096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.019261 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.019338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] token.caching = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.019498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.019574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.019752 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.019837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.019990 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.020066 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.020227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.020303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.020475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.020621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.020777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.020897 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.021027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.021170 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.021296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.021422 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.021616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.021696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.021850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.021928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.022091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.022229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.022356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.022463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.022589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.022713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.022834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.022922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.023065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.023142 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.023318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.023396 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.connection = **** {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.023611 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.023697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.023861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.023940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.connection = **** {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.024100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.024186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.024343 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.024421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.024580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.024664 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.024819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.024896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.025054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.025131 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.025290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.025441 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.025516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.025586 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.025750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.025835 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.025991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.026076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.026252 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.026331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.026501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.026578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.026737 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.026826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.026979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.027056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.027211 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.027288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.027452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.027538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.027682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.027759 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.027850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81657) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 24 03:25:23.027926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.028149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 24 03:25:23.028322 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81658) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 24 03:25:23.065789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.066014 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.066280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.066634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.066870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.067100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.076076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.076216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.108640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.108780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.108881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.108960 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.165985 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.166328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.219463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.219701 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.221199 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.221432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.383809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.384039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.384260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.384415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.384504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.384645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.384728 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.384863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.384945 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.385077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.385290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.385501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.434303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.434496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.434726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.434953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.435177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.435399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.435618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.437071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.437303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.437534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.437757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.437976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.438243 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.438448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.447581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.449432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.461450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.461588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.461670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.461746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.461892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.466833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.466922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.466997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.467149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.467236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.467326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.467402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.467474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81658) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.467546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81657) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 24 03:25:23.482934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.483141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 24 03:25:23.483481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.483681 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 24 03:25:23.483947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.484032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.484205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 24 03:25:23.484368 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.484574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.484734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.484817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 24 03:25:23.484997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.485078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.485297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.485821 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.486088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.494686 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.494963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.495235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.495519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.496198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.496483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.496760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.497048 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.502587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.504245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.505640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.505923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.506501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.507126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.507126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.507421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.507783 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.507978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.508087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.508260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.508342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.508545 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.508637 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.508832 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.508921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.509098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.509372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.509649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.510037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.510350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.510626 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.532277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.533760 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.538386 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.538578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.538578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.538578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.539327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.539614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.539901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.541634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.541920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.542226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.546279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.546568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.548723 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.549013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.550814 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.551108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.551394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.551676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.551976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.552253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.552521 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.552785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.553148 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.553439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.553724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.554014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.554321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.554605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.554882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.555265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.570449 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.570732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.573173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.573460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.574631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.577392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.577832 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.582145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.582259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.582259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.582469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.585953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.586239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.589117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.589462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.589767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.590058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.590379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.590664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.590959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.592514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.592817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.593108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.593396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.593721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.594096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.606895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.610854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.610854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.610854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.611054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.611054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.611054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.611054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.614783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.616302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.616581 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.618262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.619282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.619567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.619786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.619870 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.620067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.620159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.620414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.622776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.623143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.623444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.623816 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.624101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.632592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.632876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.633161 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.633444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.633728 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.634006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.634329 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.636885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.637172 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.637458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.637750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.638034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.638338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.638625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.650832 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.654137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.654991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.657108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.658042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.658675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.660149 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.660436 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.660729 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.660952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.662622 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.664098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.664382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.664681 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.667396 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.668849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.671227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.671876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.672166 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.672451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.672706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.672808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.675624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.675922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.676212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.676496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.680667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.680955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.681235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.684537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.684767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.685067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.688166 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81657) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.688452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.688733 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.689010 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.689287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.689568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.689845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.690122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.690427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.690707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.690989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.691252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.691525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.691795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81657) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.692076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81658) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 24 03:25:23.692949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.693237 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.693513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.693787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.694063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.694376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.694656 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.694937 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.695219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.695495 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.695759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.696052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.696315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81658) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 24 03:25:23.727119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55b7936b89c0 pid: 81657 (default app) Aug 24 03:25:23.730691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d05813-fe66-42a1-96fd-8a0aab7abb4e None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:23.730979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d05813-fe66-42a1-96fd-8a0aab7abb4e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:23.731250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d05813-fe66-42a1-96fd-8a0aab7abb4e None None] PATH_INFO: `/v3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:23.731750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55b7936b89c0 pid: 81658 (default app) Aug 24 03:25:23.735460 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1/1] 10.222.0.65 () {50 vars in 791 bytes} [Wed Aug 24 03:25:23 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:31.448899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607b51f5-415e-4498-8ea6-9c3dc89580c1 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:31.449369 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607b51f5-415e-4498-8ea6-9c3dc89580c1 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:31.449452 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607b51f5-415e-4498-8ea6-9c3dc89580c1 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:31.453690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1/2] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:31 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:31.460771 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a26b02-9b34-43d4-9451-3351c207586f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:31.461169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a26b02-9b34-43d4-9451-3351c207586f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:31.461323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a26b02-9b34-43d4-9451-3351c207586f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:31.533279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: INFO dbcounter [None req-03a26b02-9b34-43d4-9451-3351c207586f None None] Registered counter for database keystone Aug 24 03:25:31.555022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG oslo_db.sqlalchemy.engines [None req-03a26b02-9b34-43d4-9451-3351c207586f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81657) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 24 03:25:31.561364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writer thread running {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 24 03:25:31.762064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG passlib.handlers.bcrypt [None req-03a26b02-9b34-43d4-9451-3351c207586f None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=81657) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 24 03:25:31.762064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG passlib.handlers.bcrypt [None req-03a26b02-9b34-43d4-9451-3351c207586f None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81657) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 24 03:25:31.801240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-03a26b02-9b34-43d4-9451-3351c207586f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:31.803106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-03a26b02-9b34-43d4-9451-3351c207586f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:31.933478 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 2/3] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:31 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 474 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 24 03:25:32.415434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607b51f5-415e-4498-8ea6-9c3dc89580c1 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:32.484670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: INFO dbcounter [None req-607b51f5-415e-4498-8ea6-9c3dc89580c1 None None] Registered counter for database keystone Aug 24 03:25:32.504125 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG oslo_db.sqlalchemy.engines [None req-607b51f5-415e-4498-8ea6-9c3dc89580c1 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81658) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 24 03:25:32.511424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writer thread running {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 24 03:25:32.630218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607b51f5-415e-4498-8ea6-9c3dc89580c1 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:32.634780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6930971-2465-494a-9014-69d77c7a63e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:32.635398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6930971-2465-494a-9014-69d77c7a63e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:32.635672 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6930971-2465-494a-9014-69d77c7a63e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:32.635935 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6930971-2465-494a-9014-69d77c7a63e7 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:32.643820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6930971-2465-494a-9014-69d77c7a63e7 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:32.696218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: WARNING keystone.common.rbac_enforcer.enforcer [None req-a6930971-2465-494a-9014-69d77c7a63e7 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 24 03:25:32.696505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6930971-2465-494a-9014-69d77c7a63e7 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:32.705797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 2/4] 10.222.0.65 () {58 vars in 1324 bytes} [Wed Aug 24 03:25:32 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 24 03:25:32.712250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a26b02-9b34-43d4-9451-3351c207586f None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:32.727189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a26b02-9b34-43d4-9451-3351c207586f None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:32.731571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f07ce7-6da1-486a-b4f9-e628c4578182 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:32.732656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f07ce7-6da1-486a-b4f9-e628c4578182 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:32.732938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f07ce7-6da1-486a-b4f9-e628c4578182 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:32.733209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f07ce7-6da1-486a-b4f9-e628c4578182 admin admin] PATH_INFO: `/v3/projects/b8d28080f87c4783b9ef971b31a0d51a` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:32.734794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f07ce7-6da1-486a-b4f9-e628c4578182 admin admin] RBAC: Authorizing `identity:get_project(project_id=b8d28080f87c4783b9ef971b31a0d51a)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:32.790824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: WARNING keystone.common.rbac_enforcer.enforcer [None req-81f07ce7-6da1-486a-b4f9-e628c4578182 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 24 03:25:32.791168 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f07ce7-6da1-486a-b4f9-e628c4578182 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:32.795657 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 3/5] 10.222.0.65 () {56 vars in 1207 bytes} [Wed Aug 24 03:25:32 2022] GET /identity/v3/projects/b8d28080f87c4783b9ef971b31a0d51a => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:34.670805 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fb5697-0ca3-4120-ab44-87aff9582363 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:34.671315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fb5697-0ca3-4120-ab44-87aff9582363 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:34.671315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fb5697-0ca3-4120-ab44-87aff9582363 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:34.675383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 3/6] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:34 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:34.681249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796135e7-1073-45e0-9b60-67d19040ce5a None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:34.681665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796135e7-1073-45e0-9b60-67d19040ce5a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:34.681783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796135e7-1073-45e0-9b60-67d19040ce5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:34.702516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-796135e7-1073-45e0-9b60-67d19040ce5a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:34.704034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-796135e7-1073-45e0-9b60-67d19040ce5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:34.712078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 4/7] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:34 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 24 03:25:35.000447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fb5697-0ca3-4120-ab44-87aff9582363 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:35.013567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fb5697-0ca3-4120-ab44-87aff9582363 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:35.017758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001f85af-8a4d-4830-b7e2-cc3d3d51a838 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:35.018384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001f85af-8a4d-4830-b7e2-cc3d3d51a838 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:35.018664 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001f85af-8a4d-4830-b7e2-cc3d3d51a838 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:35.018933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001f85af-8a4d-4830-b7e2-cc3d3d51a838 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:35.027301 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001f85af-8a4d-4830-b7e2-cc3d3d51a838 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:35.029064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001f85af-8a4d-4830-b7e2-cc3d3d51a838 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:35.037845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 4/8] 10.222.0.65 () {58 vars in 1324 bytes} [Wed Aug 24 03:25:34 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:25:35.043304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796135e7-1073-45e0-9b60-67d19040ce5a None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:35.052464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796135e7-1073-45e0-9b60-67d19040ce5a None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:35.056978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360ddfdc-2ac1-48e9-98ef-cfdf5830f4ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:35.057619 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360ddfdc-2ac1-48e9-98ef-cfdf5830f4ee admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:35.057905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360ddfdc-2ac1-48e9-98ef-cfdf5830f4ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:35.058215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360ddfdc-2ac1-48e9-98ef-cfdf5830f4ee admin admin] PATH_INFO: `/v3/users/c73de3f5e3b94a3182c45d2476903a00` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:35.059581 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360ddfdc-2ac1-48e9-98ef-cfdf5830f4ee admin admin] RBAC: Authorizing `identity:get_user(user_id=c73de3f5e3b94a3182c45d2476903a00)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:35.061399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360ddfdc-2ac1-48e9-98ef-cfdf5830f4ee admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:35.062973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 5/9] 10.222.0.65 () {56 vars in 1198 bytes} [Wed Aug 24 03:25:35 2022] GET /identity/v3/users/c73de3f5e3b94a3182c45d2476903a00 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:37.058889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc669e9-000f-47d5-acd2-9e45512f7da8 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:37.059426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc669e9-000f-47d5-acd2-9e45512f7da8 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:37.059426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc669e9-000f-47d5-acd2-9e45512f7da8 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:37.060749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 5/10] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:37.066014 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4ba6e1-9e27-461c-a60c-dc7830e2e293 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:37.066464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4ba6e1-9e27-461c-a60c-dc7830e2e293 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:37.066590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4ba6e1-9e27-461c-a60c-dc7830e2e293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:37.071991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a593297-c96b-4995-9d0f-84f1ad9e1654 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:37.072271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a593297-c96b-4995-9d0f-84f1ad9e1654 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:37.072538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a593297-c96b-4995-9d0f-84f1ad9e1654 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:37.073793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 6/11] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:25:37.078702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533401ae-5775-4989-b0c6-025f4be0ab49 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:37.078986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533401ae-5775-4989-b0c6-025f4be0ab49 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:37.079260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533401ae-5775-4989-b0c6-025f4be0ab49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:37.085820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-2d4ba6e1-9e27-461c-a60c-dc7830e2e293 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:37.087050 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-2d4ba6e1-9e27-461c-a60c-dc7830e2e293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:37.121170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG passlib.handlers.bcrypt [None req-533401ae-5775-4989-b0c6-025f4be0ab49 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=81658) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 24 03:25:37.121609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG passlib.handlers.bcrypt [None req-533401ae-5775-4989-b0c6-025f4be0ab49 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81658) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 24 03:25:37.165735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 6/12] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:37.168303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-533401ae-5775-4989-b0c6-025f4be0ab49 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:37.169914 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-533401ae-5775-4989-b0c6-025f4be0ab49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:37.267827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 7/13] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:25:37.436273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4ba6e1-9e27-461c-a60c-dc7830e2e293 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:37.460616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:37.537749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533401ae-5775-4989-b0c6-025f4be0ab49 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:37.541766 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4ba6e1-9e27-461c-a60c-dc7830e2e293 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:37.575815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4ffe92-b4ca-442c-bc3b-2bb815e067bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:37.576448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4ffe92-b4ca-442c-bc3b-2bb815e067bd None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:37.576730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4ffe92-b4ca-442c-bc3b-2bb815e067bd None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:37.577001 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4ffe92-b4ca-442c-bc3b-2bb815e067bd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:37.588565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4ffe92-b4ca-442c-bc3b-2bb815e067bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-5c4ffe92-b4ca-442c-bc3b-2bb815e067bd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:25:37.596962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:37.598042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:25:37.599100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 7/14] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:25:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:25:37.604401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4ffe92-b4ca-442c-bc3b-2bb815e067bd None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:37.645877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533401ae-5775-4989-b0c6-025f4be0ab49 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:37.648198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:37.683230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00194a02-479b-4654-a5b8-f0bb83a6d556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:37.684010 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00194a02-479b-4654-a5b8-f0bb83a6d556 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:37.684295 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00194a02-479b-4654-a5b8-f0bb83a6d556 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:37.684572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00194a02-479b-4654-a5b8-f0bb83a6d556 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:37.684999 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:37.703219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4ffe92-b4ca-442c-bc3b-2bb815e067bd None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:37.737486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8282ad-8cd2-47da-b8d5-12667564081b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:37.738082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8282ad-8cd2-47da-b8d5-12667564081b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:37.738394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8282ad-8cd2-47da-b8d5-12667564081b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:37.738662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8282ad-8cd2-47da-b8d5-12667564081b None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:37.739887 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8282ad-8cd2-47da-b8d5-12667564081b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:37.740301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8282ad-8cd2-47da-b8d5-12667564081b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:37.741689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8282ad-8cd2-47da-b8d5-12667564081b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:37.753098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00194a02-479b-4654-a5b8-f0bb83a6d556 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:37.753263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 8/15] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:25:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:25:37.755245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00194a02-479b-4654-a5b8-f0bb83a6d556 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:37.759676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8282ad-8cd2-47da-b8d5-12667564081b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:37.853897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 8/16] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:25:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 03:25:37.856929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8282ad-8cd2-47da-b8d5-12667564081b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:37.859459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00194a02-479b-4654-a5b8-f0bb83a6d556 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:37.862127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:37.891412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b88cbf9-6e1b-4453-be57-75bd333f9cf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:37.892033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b88cbf9-6e1b-4453-be57-75bd333f9cf2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:37.892305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b88cbf9-6e1b-4453-be57-75bd333f9cf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:37.892565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b88cbf9-6e1b-4453-be57-75bd333f9cf2 None admin] PATH_INFO: `/v3/users/c73de3f5e3b94a3182c45d2476903a00` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:37.893735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b88cbf9-6e1b-4453-be57-75bd333f9cf2 None admin] RBAC: Authorizing `identity:get_user(user_id=c73de3f5e3b94a3182c45d2476903a00)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:37.895348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b88cbf9-6e1b-4453-be57-75bd333f9cf2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:37.897242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 9/17] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:25:37 2022] GET /identity/v3/users/c73de3f5e3b94a3182c45d2476903a00 => generated 247 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:25:37.902826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b88cbf9-6e1b-4453-be57-75bd333f9cf2 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:37.917709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:37.958526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00194a02-479b-4654-a5b8-f0bb83a6d556 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:37.992208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1497bcff-0d05-4188-80f2-8199c350b01d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:37.993063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1497bcff-0d05-4188-80f2-8199c350b01d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:37.993343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1497bcff-0d05-4188-80f2-8199c350b01d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:37.993615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1497bcff-0d05-4188-80f2-8199c350b01d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:38.004273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b88cbf9-6e1b-4453-be57-75bd333f9cf2 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:38.012097 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1497bcff-0d05-4188-80f2-8199c350b01d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:38.013559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1497bcff-0d05-4188-80f2-8199c350b01d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-1497bcff-0d05-4188-80f2-8199c350b01d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:25:38.031371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:38.032270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:25:38.033230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:25:38.033230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:25:38.033230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:25:38.033230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 9/18] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:25:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:25:38.038788 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1497bcff-0d05-4188-80f2-8199c350b01d None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:38.040209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173edbd6-81a8-4bef-8dfd-aed8acc1fc29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:38.040848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173edbd6-81a8-4bef-8dfd-aed8acc1fc29 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:38.041132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173edbd6-81a8-4bef-8dfd-aed8acc1fc29 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:38.041443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173edbd6-81a8-4bef-8dfd-aed8acc1fc29 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:38.042671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173edbd6-81a8-4bef-8dfd-aed8acc1fc29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:38.044127 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173edbd6-81a8-4bef-8dfd-aed8acc1fc29 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:38.045673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 10/19] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:25:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:25:38.051234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173edbd6-81a8-4bef-8dfd-aed8acc1fc29 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:38.093885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:38.132896 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:38.139025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1497bcff-0d05-4188-80f2-8199c350b01d None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:38.151438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173edbd6-81a8-4bef-8dfd-aed8acc1fc29 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:38.172673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c36ae36-11aa-4d8a-983b-d9b008eb4fd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:38.173275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c36ae36-11aa-4d8a-983b-d9b008eb4fd4 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:38.173552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c36ae36-11aa-4d8a-983b-d9b008eb4fd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:38.173818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c36ae36-11aa-4d8a-983b-d9b008eb4fd4 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:38.174492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c36ae36-11aa-4d8a-983b-d9b008eb4fd4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:38.174791 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c36ae36-11aa-4d8a-983b-d9b008eb4fd4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:38.176177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c36ae36-11aa-4d8a-983b-d9b008eb4fd4 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:38.186379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2580e0-d46c-49f6-9087-9b4312203eca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:38.186994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2580e0-d46c-49f6-9087-9b4312203eca None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:38.187245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2580e0-d46c-49f6-9087-9b4312203eca None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:38.187507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2580e0-d46c-49f6-9087-9b4312203eca None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:38.188224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2580e0-d46c-49f6-9087-9b4312203eca None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.domain.id=default, user.id=c73de3f5e3b94a3182c45d2476903a00) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:38.188525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2580e0-d46c-49f6-9087-9b4312203eca None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:38.189943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2580e0-d46c-49f6-9087-9b4312203eca None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:38.195759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 10/20] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:25:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:25:38.202210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 11/21] 10.222.0.65 () {56 vars in 1322 bytes} [Wed Aug 24 03:25:38 2022] GET /identity/v3/role_assignments?user.id=c73de3f5e3b94a3182c45d2476903a00&scope.domain.id=default&role.id=43ca25467bc6484da3a50508efb79230 => generated 233 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:40.266607 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1736225b-b42e-4e96-9b8f-7e11355725be None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:40.267147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1736225b-b42e-4e96-9b8f-7e11355725be None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:40.267147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1736225b-b42e-4e96-9b8f-7e11355725be None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:40.268497 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 11/22] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:40.283632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c393d4-f0cb-49f9-acb9-16b44550c1b2 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:40.284094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c393d4-f0cb-49f9-acb9-16b44550c1b2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:40.284165 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c393d4-f0cb-49f9-acb9-16b44550c1b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:40.311195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e9c393d4-f0cb-49f9-acb9-16b44550c1b2 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:40.312427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e9c393d4-f0cb-49f9-acb9-16b44550c1b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:40.378713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844624b9-bf7a-47cb-8536-9c3252e638b1 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:40.379026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844624b9-bf7a-47cb-8536-9c3252e638b1 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:40.379276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844624b9-bf7a-47cb-8536-9c3252e638b1 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:40.381871 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 12/23] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:40.388558 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ccdf01-0627-4a13-9559-eecf4d9a5772 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:40.388843 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ccdf01-0627-4a13-9559-eecf4d9a5772 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:40.389117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ccdf01-0627-4a13-9559-eecf4d9a5772 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:40.401179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 12/24] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:40.416529 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-60ccdf01-0627-4a13-9559-eecf4d9a5772 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:40.417766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-60ccdf01-0627-4a13-9559-eecf4d9a5772 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:40.512767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 13/25] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:25:40.718667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23599d9b-7be0-4026-967d-81db85172d7f None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:40.719367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23599d9b-7be0-4026-967d-81db85172d7f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:40.719834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23599d9b-7be0-4026-967d-81db85172d7f None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:40.721351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 13/26] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:40.727341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5b7d6c-6f71-41a2-99b8-83d6a912c453 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:40.727633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5b7d6c-6f71-41a2-99b8-83d6a912c453 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:40.727911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5b7d6c-6f71-41a2-99b8-83d6a912c453 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:40.750362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-5a5b7d6c-6f71-41a2-99b8-83d6a912c453 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:40.751561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5a5b7d6c-6f71-41a2-99b8-83d6a912c453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:40.801971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23599d9b-7be0-4026-967d-81db85172d7f None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:40.808588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:40.849681 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 14/27] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:40.950482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:40.982924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23599d9b-7be0-4026-967d-81db85172d7f None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:41.027982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a656d5cd-fd7a-41e9-a8f1-d7e31f03cbd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:41.028625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a656d5cd-fd7a-41e9-a8f1-d7e31f03cbd0 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:41.028908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a656d5cd-fd7a-41e9-a8f1-d7e31f03cbd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:41.029179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a656d5cd-fd7a-41e9-a8f1-d7e31f03cbd0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:41.043165 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a656d5cd-fd7a-41e9-a8f1-d7e31f03cbd0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-a656d5cd-fd7a-41e9-a8f1-d7e31f03cbd0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:25:41.048073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:41.048899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:25:41.049638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 14/28] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:25:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:25:41.056716 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5b7d6c-6f71-41a2-99b8-83d6a912c453 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:41.076898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a656d5cd-fd7a-41e9-a8f1-d7e31f03cbd0 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:41.245456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a656d5cd-fd7a-41e9-a8f1-d7e31f03cbd0 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:41.264417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5b7d6c-6f71-41a2-99b8-83d6a912c453 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:41.303189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffda7cc-7804-4610-8054-5918addb66e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:41.304996 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffda7cc-7804-4610-8054-5918addb66e8 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:41.305876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffda7cc-7804-4610-8054-5918addb66e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:41.306683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffda7cc-7804-4610-8054-5918addb66e8 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:41.308036 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffda7cc-7804-4610-8054-5918addb66e8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:41.308844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce38ad24-ad38-48b0-a922-1658ab963290 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:41.309589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffda7cc-7804-4610-8054-5918addb66e8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:41.310382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce38ad24-ad38-48b0-a922-1658ab963290 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:41.311909 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffda7cc-7804-4610-8054-5918addb66e8 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:41.314287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce38ad24-ad38-48b0-a922-1658ab963290 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:41.314287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce38ad24-ad38-48b0-a922-1658ab963290 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:41.318423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:41.333829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 15/29] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:25:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:41.335253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3680c982-5016-4680-8bb8-43536d1bf87c None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:41.335539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3680c982-5016-4680-8bb8-43536d1bf87c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:41.335822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3680c982-5016-4680-8bb8-43536d1bf87c None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:41.337246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 16/30] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:41.340563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce38ad24-ad38-48b0-a922-1658ab963290 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:41.342931 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce38ad24-ad38-48b0-a922-1658ab963290 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:41.346543 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3680c982-5016-4680-8bb8-43536d1bf87c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-ce38ad24-ad38-48b0-a922-1658ab963290 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:25:41.361639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:41.362567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:25:41.363372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:25:41.363372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:25:41.363372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:25:41.365738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 15/31] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:25:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:25:41.367359 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6ba0fe-a07e-4d2b-bd3f-aef00baac5f8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:41.367846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6ba0fe-a07e-4d2b-bd3f-aef00baac5f8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:41.368361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6ba0fe-a07e-4d2b-bd3f-aef00baac5f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:41.386786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-4e6ba0fe-a07e-4d2b-bd3f-aef00baac5f8 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:41.388086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4e6ba0fe-a07e-4d2b-bd3f-aef00baac5f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:41.392700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:41.466660 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 16/32] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:41.467811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6ba0fe-a07e-4d2b-bd3f-aef00baac5f8 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:41.469559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3680c982-5016-4680-8bb8-43536d1bf87c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:41.506216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a99f18b-3a87-42ac-bbb5-f93314ecaf00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:41.506764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a99f18b-3a87-42ac-bbb5-f93314ecaf00 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:41.507045 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a99f18b-3a87-42ac-bbb5-f93314ecaf00 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:41.507320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a99f18b-3a87-42ac-bbb5-f93314ecaf00 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:41.508068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a99f18b-3a87-42ac-bbb5-f93314ecaf00 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:41.509750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a99f18b-3a87-42ac-bbb5-f93314ecaf00 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:41.579456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6ba0fe-a07e-4d2b-bd3f-aef00baac5f8 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:41.598768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:41.617457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3310fe-f0ce-4fbe-8d00-e35855b6107d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:41.618378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3310fe-f0ce-4fbe-8d00-e35855b6107d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:41.618850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3310fe-f0ce-4fbe-8d00-e35855b6107d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:41.619291 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3310fe-f0ce-4fbe-8d00-e35855b6107d None admin] PATH_INFO: `/v3/users/c73de3f5e3b94a3182c45d2476903a00` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:41.620732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3310fe-f0ce-4fbe-8d00-e35855b6107d None admin] RBAC: Authorizing `identity:get_user(user_id=c73de3f5e3b94a3182c45d2476903a00)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:41.622362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3310fe-f0ce-4fbe-8d00-e35855b6107d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:41.628022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 17/33] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:25:41 2022] GET /identity/v3/users/c73de3f5e3b94a3182c45d2476903a00 => generated 247 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:41.628022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3310fe-f0ce-4fbe-8d00-e35855b6107d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:41.729335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3310fe-f0ce-4fbe-8d00-e35855b6107d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:41.764626 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8f5c0c-bb3d-4a29-9adc-edb7e7e4a645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:41.765160 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8f5c0c-bb3d-4a29-9adc-edb7e7e4a645 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:41.765440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8f5c0c-bb3d-4a29-9adc-edb7e7e4a645 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:41.765712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8f5c0c-bb3d-4a29-9adc-edb7e7e4a645 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:41.766377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8f5c0c-bb3d-4a29-9adc-edb7e7e4a645 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:41.766687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8f5c0c-bb3d-4a29-9adc-edb7e7e4a645 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:41.768192 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8f5c0c-bb3d-4a29-9adc-edb7e7e4a645 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:41.786945 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 18/34] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:25:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:41.787965 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8f5c0c-bb3d-4a29-9adc-edb7e7e4a645 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:41.827800 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 17/35] 10.222.0.65 () {60 vars in 1131 bytes} [Wed Aug 24 03:25:41 2022] POST /identity/v3/roles => generated 225 bytes in 483 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:25:41.828840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a99f18b-3a87-42ac-bbb5-f93314ecaf00 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:41.846934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:41.959350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8f5c0c-bb3d-4a29-9adc-edb7e7e4a645 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:42.019948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5587b752-131f-4839-af8c-aec8af954948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:42.021444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5587b752-131f-4839-af8c-aec8af954948 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:42.022282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:42.023011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5587b752-131f-4839-af8c-aec8af954948 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:42.023296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5587b752-131f-4839-af8c-aec8af954948 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:42.024462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5587b752-131f-4839-af8c-aec8af954948 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:42.026556 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5587b752-131f-4839-af8c-aec8af954948 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:42.027754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a99f18b-3a87-42ac-bbb5-f93314ecaf00 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:42.028326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 19/36] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:25:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:42.028861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5587b752-131f-4839-af8c-aec8af954948 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:42.068142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89448dad-370d-4433-8465-c3a09adc4f78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:42.068920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89448dad-370d-4433-8465-c3a09adc4f78 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:42.069207 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89448dad-370d-4433-8465-c3a09adc4f78 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:42.069484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89448dad-370d-4433-8465-c3a09adc4f78 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:42.070505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89448dad-370d-4433-8465-c3a09adc4f78 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:42.072303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89448dad-370d-4433-8465-c3a09adc4f78 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:42.149573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5587b752-131f-4839-af8c-aec8af954948 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:42.163317 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:42.184353 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8290afd9-5a94-4e69-9f7b-b2bdd4efd52a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:42.184959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8290afd9-5a94-4e69-9f7b-b2bdd4efd52a None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:42.185238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8290afd9-5a94-4e69-9f7b-b2bdd4efd52a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:42.185508 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8290afd9-5a94-4e69-9f7b-b2bdd4efd52a None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:42.186195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8290afd9-5a94-4e69-9f7b-b2bdd4efd52a None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:42.187632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8290afd9-5a94-4e69-9f7b-b2bdd4efd52a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:42.213769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 18/37] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:25:41 2022] POST /identity/v3/projects => generated 301 bytes in 385 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:25:42.216271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89448dad-370d-4433-8465-c3a09adc4f78 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:42.331629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89448dad-370d-4433-8465-c3a09adc4f78 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:42.372312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb902b1-d0c9-4a40-85cf-467f446a1821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:42.373037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb902b1-d0c9-4a40-85cf-467f446a1821 None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:42.373325 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb902b1-d0c9-4a40-85cf-467f446a1821 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:42.373606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb902b1-d0c9-4a40-85cf-467f446a1821 None admin] PATH_INFO: `/v3/domains/default/users/c73de3f5e3b94a3182c45d2476903a00/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:42.377151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb902b1-d0c9-4a40-85cf-467f446a1821 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=c73de3f5e3b94a3182c45d2476903a00, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:42.378653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb902b1-d0c9-4a40-85cf-467f446a1821 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:42.394269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 20/38] 10.222.0.65 () {60 vars in 1131 bytes} [Wed Aug 24 03:25:42 2022] POST /identity/v3/roles => generated 219 bytes in 366 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:25:42.397359 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7fde4-3423-455b-b1e9-cb43dbda7e59 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:42.397359 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7fde4-3423-455b-b1e9-cb43dbda7e59 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:42.397359 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7fde4-3423-455b-b1e9-cb43dbda7e59 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:42.398086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 21/39] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:42.399289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a8bdf9-adb8-414c-a764-708512e7552d None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:42.399643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a8bdf9-adb8-414c-a764-708512e7552d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:42.399927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a8bdf9-adb8-414c-a764-708512e7552d None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:42.401437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 22/40] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:42.406053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f06da5-dc90-407e-9a7d-a63e79a4761d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:42.406244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f06da5-dc90-407e-9a7d-a63e79a4761d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:42.406981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f06da5-dc90-407e-9a7d-a63e79a4761d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:42.432170 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-98f06da5-dc90-407e-9a7d-a63e79a4761d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:42.433760 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-98f06da5-dc90-407e-9a7d-a63e79a4761d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:42.471140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 19/41] 10.222.0.65 () {58 vars in 1361 bytes} [Wed Aug 24 03:25:42 2022] PUT /identity/v3/domains/default/users/c73de3f5e3b94a3182c45d2476903a00/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:25:42.472388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a2306f-a225-4f07-81af-8ee90c4ccb24 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:42.472676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a2306f-a225-4f07-81af-8ee90c4ccb24 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:42.472956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a2306f-a225-4f07-81af-8ee90c4ccb24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:42.495020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-45a2306f-a225-4f07-81af-8ee90c4ccb24 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:42.496889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-45a2306f-a225-4f07-81af-8ee90c4ccb24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:42.522583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:42.594613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 20/42] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:42.635481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 23/43] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:25:42.876862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a2306f-a225-4f07-81af-8ee90c4ccb24 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:42.903381 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f06da5-dc90-407e-9a7d-a63e79a4761d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:42.991271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a2306f-a225-4f07-81af-8ee90c4ccb24 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:43.005152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f06da5-dc90-407e-9a7d-a63e79a4761d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:43.006970 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:43.034434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258ea322-431f-4ac4-a3b1-9dd0eb4ccd23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:43.035336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258ea322-431f-4ac4-a3b1-9dd0eb4ccd23 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:43.036124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258ea322-431f-4ac4-a3b1-9dd0eb4ccd23 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:43.036403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258ea322-431f-4ac4-a3b1-9dd0eb4ccd23 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:43.037490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258ea322-431f-4ac4-a3b1-9dd0eb4ccd23 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:43.038978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258ea322-431f-4ac4-a3b1-9dd0eb4ccd23 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:43.039614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327948dd-21c6-48bb-aacd-1d80b4bcc2fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:43.040228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327948dd-21c6-48bb-aacd-1d80b4bcc2fe None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:43.040721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327948dd-21c6-48bb-aacd-1d80b4bcc2fe None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:43.040818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 21/44] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:25:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:43.040894 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327948dd-21c6-48bb-aacd-1d80b4bcc2fe None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:43.041442 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327948dd-21c6-48bb-aacd-1d80b4bcc2fe None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:43.042996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327948dd-21c6-48bb-aacd-1d80b4bcc2fe None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:43.045942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258ea322-431f-4ac4-a3b1-9dd0eb4ccd23 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:43.057709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:43.143212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258ea322-431f-4ac4-a3b1-9dd0eb4ccd23 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:43.156728 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 24/45] 10.222.0.65 () {60 vars in 1131 bytes} [Wed Aug 24 03:25:42 2022] POST /identity/v3/roles => generated 223 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:25:43.181362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ebc70b-c528-4337-8d59-61b378e8ba36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:43.182003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ebc70b-c528-4337-8d59-61b378e8ba36 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:43.182409 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ebc70b-c528-4337-8d59-61b378e8ba36 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:43.182812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ebc70b-c528-4337-8d59-61b378e8ba36 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:43.183491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ebc70b-c528-4337-8d59-61b378e8ba36 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:43.185001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ebc70b-c528-4337-8d59-61b378e8ba36 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:43.282187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 22/46] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:25:43 2022] POST /identity/v3/projects => generated 312 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:25:44.427367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab088b2-525b-44bb-83bd-f2f7d948318b None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:44.427778 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab088b2-525b-44bb-83bd-f2f7d948318b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:44.427922 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab088b2-525b-44bb-83bd-f2f7d948318b None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:44.429279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 25/47] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:44.434631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a913f-4807-4599-ab0f-6d0cbe05941c None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:44.435022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a913f-4807-4599-ab0f-6d0cbe05941c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:44.435184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a913f-4807-4599-ab0f-6d0cbe05941c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:44.452922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d29a913f-4807-4599-ab0f-6d0cbe05941c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:44.453979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d29a913f-4807-4599-ab0f-6d0cbe05941c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:44.508567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:44.519281 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 23/48] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:44.789744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab088b2-525b-44bb-83bd-f2f7d948318b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:44.889242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab088b2-525b-44bb-83bd-f2f7d948318b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:44.921339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838bce2e-0fcf-47a9-96e9-3f893b6e325b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:44.921963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838bce2e-0fcf-47a9-96e9-3f893b6e325b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:44.922286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838bce2e-0fcf-47a9-96e9-3f893b6e325b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:44.922561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838bce2e-0fcf-47a9-96e9-3f893b6e325b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:44.933344 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838bce2e-0fcf-47a9-96e9-3f893b6e325b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-838bce2e-0fcf-47a9-96e9-3f893b6e325b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:25:44.938079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:44.939088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:25:44.943282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 26/49] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:25:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:25:44.945809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29a913f-4807-4599-ab0f-6d0cbe05941c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:45.042501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29a913f-4807-4599-ab0f-6d0cbe05941c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:45.076107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4853bb-30af-48a9-a731-937180d891e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:45.076714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4853bb-30af-48a9-a731-937180d891e7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:45.076995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4853bb-30af-48a9-a731-937180d891e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:45.077267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4853bb-30af-48a9-a731-937180d891e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:45.077933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4853bb-30af-48a9-a731-937180d891e7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:45.078266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4853bb-30af-48a9-a731-937180d891e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:45.079656 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4853bb-30af-48a9-a731-937180d891e7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:45.087858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 24/50] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:25:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:45.093444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838bce2e-0fcf-47a9-96e9-3f893b6e325b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:45.118259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:45.166332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8d1336-3b6f-4136-9ac6-d9d33ee4dd01 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:45.166618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8d1336-3b6f-4136-9ac6-d9d33ee4dd01 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:45.166892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8d1336-3b6f-4136-9ac6-d9d33ee4dd01 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:45.168239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 25/51] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:25:45.173175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610792fd-4518-4daa-af56-475abdef89e3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:45.173462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610792fd-4518-4daa-af56-475abdef89e3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:45.173739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610792fd-4518-4daa-af56-475abdef89e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:45.191561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-610792fd-4518-4daa-af56-475abdef89e3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:45.193571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-610792fd-4518-4daa-af56-475abdef89e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:45.194587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838bce2e-0fcf-47a9-96e9-3f893b6e325b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:45.228155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97462425-358f-4a1c-b9c1-0119d9c9a955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:45.228786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97462425-358f-4a1c-b9c1-0119d9c9a955 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:45.229072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97462425-358f-4a1c-b9c1-0119d9c9a955 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:45.229351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97462425-358f-4a1c-b9c1-0119d9c9a955 None admin] PATH_INFO: `/v3/users/c73de3f5e3b94a3182c45d2476903a00` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:45.230578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97462425-358f-4a1c-b9c1-0119d9c9a955 None admin] RBAC: Authorizing `identity:get_user(user_id=c73de3f5e3b94a3182c45d2476903a00)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:45.232047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97462425-358f-4a1c-b9c1-0119d9c9a955 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:45.233846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 27/52] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:25:45 2022] GET /identity/v3/users/c73de3f5e3b94a3182c45d2476903a00 => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:45.239789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97462425-358f-4a1c-b9c1-0119d9c9a955 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:45.242387 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:45.264929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 26/53] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:25:45.330337 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:45.340003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97462425-358f-4a1c-b9c1-0119d9c9a955 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:45.373097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13984fb-c0c0-437f-bf65-b10a7e6310fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:45.373718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13984fb-c0c0-437f-bf65-b10a7e6310fa None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:45.373995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13984fb-c0c0-437f-bf65-b10a7e6310fa None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:45.374290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13984fb-c0c0-437f-bf65-b10a7e6310fa None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:45.375398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13984fb-c0c0-437f-bf65-b10a7e6310fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:45.376886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13984fb-c0c0-437f-bf65-b10a7e6310fa None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:45.379851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 28/54] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:25:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:25:45.384652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610792fd-4518-4daa-af56-475abdef89e3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:45.482694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610792fd-4518-4daa-af56-475abdef89e3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:45.515729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ccfe77-07b8-4dff-9118-7f98044ef1fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:45.516325 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ccfe77-07b8-4dff-9118-7f98044ef1fd None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:45.516601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ccfe77-07b8-4dff-9118-7f98044ef1fd None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:45.516869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ccfe77-07b8-4dff-9118-7f98044ef1fd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:45.517588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ccfe77-07b8-4dff-9118-7f98044ef1fd None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.domain.id=default, user.id=c73de3f5e3b94a3182c45d2476903a00) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:45.517888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ccfe77-07b8-4dff-9118-7f98044ef1fd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:45.519382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ccfe77-07b8-4dff-9118-7f98044ef1fd None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:45.527457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 27/55] 10.222.0.65 () {56 vars in 1322 bytes} [Wed Aug 24 03:25:45 2022] GET /identity/v3/role_assignments?user.id=c73de3f5e3b94a3182c45d2476903a00&scope.domain.id=default&role.id=43ca25467bc6484da3a50508efb79230 => generated 530 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:45.534741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13984fb-c0c0-437f-bf65-b10a7e6310fa None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:45.641709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13984fb-c0c0-437f-bf65-b10a7e6310fa None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:45.678462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005bbf2f-973e-44f9-825b-70409763b666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:45.679102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005bbf2f-973e-44f9-825b-70409763b666 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:45.679408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005bbf2f-973e-44f9-825b-70409763b666 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:45.679713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005bbf2f-973e-44f9-825b-70409763b666 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:45.680859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005bbf2f-973e-44f9-825b-70409763b666 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:45.682346 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005bbf2f-973e-44f9-825b-70409763b666 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:45.684076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 29/56] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:25:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:45.689573 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ccfe77-07b8-4dff-9118-7f98044ef1fd None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:45.738546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:45.895422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ccfe77-07b8-4dff-9118-7f98044ef1fd None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:45.927950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996c64cb-8868-47cd-b7f3-3be6f35c37e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:45.928547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996c64cb-8868-47cd-b7f3-3be6f35c37e5 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:45.928826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996c64cb-8868-47cd-b7f3-3be6f35c37e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:45.929099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996c64cb-8868-47cd-b7f3-3be6f35c37e5 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:45.929734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996c64cb-8868-47cd-b7f3-3be6f35c37e5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:45.931204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996c64cb-8868-47cd-b7f3-3be6f35c37e5 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:46.023972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 28/57] 10.222.0.65 () {60 vars in 1140 bytes} [Wed Aug 24 03:25:45 2022] POST /identity/v3/projects => generated 298 bytes in 335 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:25:47.917513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a67b5fb-d68b-42b0-ab6c-4e2b68d8cb48 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:47.918042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a67b5fb-d68b-42b0-ab6c-4e2b68d8cb48 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:47.918042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a67b5fb-d68b-42b0-ab6c-4e2b68d8cb48 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:47.919440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 30/58] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:47.924707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80b3067-d546-4c2e-ab05-1920a3a04b1d None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:47.925071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80b3067-d546-4c2e-ab05-1920a3a04b1d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:47.925258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80b3067-d546-4c2e-ab05-1920a3a04b1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:47.944114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d80b3067-d546-4c2e-ab05-1920a3a04b1d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:47.945317 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d80b3067-d546-4c2e-ab05-1920a3a04b1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:48.010368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 29/59] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:48.283298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a67b5fb-d68b-42b0-ab6c-4e2b68d8cb48 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:48.312161 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:48.385993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a67b5fb-d68b-42b0-ab6c-4e2b68d8cb48 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:48.420009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5276d27-2255-416c-acd8-ed6d7b91b13f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:48.420649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5276d27-2255-416c-acd8-ed6d7b91b13f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:48.420934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5276d27-2255-416c-acd8-ed6d7b91b13f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:48.421234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5276d27-2255-416c-acd8-ed6d7b91b13f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:48.422393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5276d27-2255-416c-acd8-ed6d7b91b13f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:48.423826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5276d27-2255-416c-acd8-ed6d7b91b13f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:48.425383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 31/60] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:25:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:48.431145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80b3067-d546-4c2e-ab05-1920a3a04b1d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:48.459351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:48.529651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80b3067-d546-4c2e-ab05-1920a3a04b1d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:48.561643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d1b3bc-207a-4288-a871-0909002c7471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:48.562209 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d1b3bc-207a-4288-a871-0909002c7471 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:48.562490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d1b3bc-207a-4288-a871-0909002c7471 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:48.562762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d1b3bc-207a-4288-a871-0909002c7471 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:48.563414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d1b3bc-207a-4288-a871-0909002c7471 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:48.564849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d1b3bc-207a-4288-a871-0909002c7471 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:48.636512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 30/61] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:25:48 2022] POST /identity/v3/users => generated 275 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:25:51.539673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3237b766-99d7-45f8-838d-e90b24796cd0 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:51.540701 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3237b766-99d7-45f8-838d-e90b24796cd0 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:51.541179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3237b766-99d7-45f8-838d-e90b24796cd0 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:51.542828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 32/62] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:51.548374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a591b39c-3352-4eda-ba4e-326921f8e0e3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:51.549170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a591b39c-3352-4eda-ba4e-326921f8e0e3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:51.549640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a591b39c-3352-4eda-ba4e-326921f8e0e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:51.579144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-a591b39c-3352-4eda-ba4e-326921f8e0e3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:51.580647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a591b39c-3352-4eda-ba4e-326921f8e0e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:51.670370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 31/63] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:25:51.854743 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b605b196-1d56-4cc4-8235-916ec56faedc None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:51.855421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b605b196-1d56-4cc4-8235-916ec56faedc None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:51.855901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b605b196-1d56-4cc4-8235-916ec56faedc None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:51.857439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 33/64] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:51.867586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59548391-586c-4c60-a938-40f1fb6a2557 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:51.868132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59548391-586c-4c60-a938-40f1fb6a2557 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:51.868612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59548391-586c-4c60-a938-40f1fb6a2557 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:51.891933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-59548391-586c-4c60-a938-40f1fb6a2557 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:51.893429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-59548391-586c-4c60-a938-40f1fb6a2557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:51.904343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:51.957282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f641136f-7bbe-460b-a3c4-377f284e7b56 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:51.957772 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f641136f-7bbe-460b-a3c4-377f284e7b56 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:51.958294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f641136f-7bbe-460b-a3c4-377f284e7b56 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:51.960091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 34/65] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:51.965770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f4bbcb-2939-4016-bdda-cc5501084d31 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:51.967902 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f4bbcb-2939-4016-bdda-cc5501084d31 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:51.967902 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f4bbcb-2939-4016-bdda-cc5501084d31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:51.993062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 32/66] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:51.994356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c9f4bbcb-2939-4016-bdda-cc5501084d31 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:51.995790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c9f4bbcb-2939-4016-bdda-cc5501084d31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:52.096129 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 35/67] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:25:52.107173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59548391-586c-4c60-a938-40f1fb6a2557 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:52.212471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59548391-586c-4c60-a938-40f1fb6a2557 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:52.237334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36653b4-a4a8-4a3d-86de-bd47b4b9fe36 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:52.237991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36653b4-a4a8-4a3d-86de-bd47b4b9fe36 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:52.238586 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36653b4-a4a8-4a3d-86de-bd47b4b9fe36 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:52.240351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 36/68] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:52.245744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ae345f-395c-4f7b-a2b6-0bdf25240926 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:52.246288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ae345f-395c-4f7b-a2b6-0bdf25240926 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:52.246932 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ae345f-395c-4f7b-a2b6-0bdf25240926 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:52.266545 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c1ae345f-395c-4f7b-a2b6-0bdf25240926 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:52.268038 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c1ae345f-395c-4f7b-a2b6-0bdf25240926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:52.283771 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c44c77e-1172-4b43-b94a-08f2f1a30df1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:52.284750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c44c77e-1172-4b43-b94a-08f2f1a30df1 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:52.285160 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c44c77e-1172-4b43-b94a-08f2f1a30df1 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:52.285494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c44c77e-1172-4b43-b94a-08f2f1a30df1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:52.291055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:52.306664 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c44c77e-1172-4b43-b94a-08f2f1a30df1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-5c44c77e-1172-4b43-b94a-08f2f1a30df1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:25:52.315839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:52.318035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:52.318836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:52.318836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:52.318836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:52.318836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:52.318836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:25:52.318836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:25:52.318836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:25:52.318836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:25:52.318836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:25:52.318836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:25:52.318836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:25:52.318836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:25:52.320719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 33/69] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:25:52 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:25:52.321847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c44c77e-1172-4b43-b94a-08f2f1a30df1 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:52.325599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:52.352561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 37/70] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:25:52.353678 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ae345f-395c-4f7b-a2b6-0bdf25240926 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:52.461951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c44c77e-1172-4b43-b94a-08f2f1a30df1 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:52.486591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ae345f-395c-4f7b-a2b6-0bdf25240926 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:52.496739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f2e0de-2d25-4dcf-821e-fdafa869a3a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:52.497370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f2e0de-2d25-4dcf-821e-fdafa869a3a8 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:52.497940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f2e0de-2d25-4dcf-821e-fdafa869a3a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:52.497940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f2e0de-2d25-4dcf-821e-fdafa869a3a8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:52.509253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f2e0de-2d25-4dcf-821e-fdafa869a3a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-44f2e0de-2d25-4dcf-821e-fdafa869a3a8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:25:52.514422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:52.515341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:25:52.516098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 34/71] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:25:52 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:25:52.517399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3a7174-0e5f-482e-b316-14e1c6ef509e None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:52.517399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3a7174-0e5f-482e-b316-14e1c6ef509e None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:52.517399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3a7174-0e5f-482e-b316-14e1c6ef509e None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:52.518591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 35/72] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:52 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:52.519206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3a7174-0e5f-482e-b316-14e1c6ef509e None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:52.531912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b721272-1184-47be-af90-b88476ee6f10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:52.532527 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b721272-1184-47be-af90-b88476ee6f10 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:52.532815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b721272-1184-47be-af90-b88476ee6f10 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:52.533132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b721272-1184-47be-af90-b88476ee6f10 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:52.533807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b721272-1184-47be-af90-b88476ee6f10 None admin] RBAC: Adding query filter params (name=member) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:52.534111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b721272-1184-47be-af90-b88476ee6f10 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:52.535654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b721272-1184-47be-af90-b88476ee6f10 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:52.544592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 38/73] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:25:52 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:52.546052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b721272-1184-47be-af90-b88476ee6f10 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:52.567724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:52.589479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:52.628233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3a7174-0e5f-482e-b316-14e1c6ef509e None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:52.652394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b721272-1184-47be-af90-b88476ee6f10 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:52.663199 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad9da3f-f739-47de-93ec-ea5c57c7e140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:52.663833 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad9da3f-f739-47de-93ec-ea5c57c7e140 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:52.664113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad9da3f-f739-47de-93ec-ea5c57c7e140 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:52.664389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad9da3f-f739-47de-93ec-ea5c57c7e140 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:52.674444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad9da3f-f739-47de-93ec-ea5c57c7e140 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-2ad9da3f-f739-47de-93ec-ea5c57c7e140 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:25:52.680126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:52.680972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 36/74] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:25:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:25:52.682455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d077b0-53de-4210-a1e9-c6eafdb4864c None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:52.683208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d077b0-53de-4210-a1e9-c6eafdb4864c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:52.683208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d077b0-53de-4210-a1e9-c6eafdb4864c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:52.687004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416b301b-d659-4224-be41-86002a79eeb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:52.687624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416b301b-d659-4224-be41-86002a79eeb0 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:52.687906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416b301b-d659-4224-be41-86002a79eeb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:52.688181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416b301b-d659-4224-be41-86002a79eeb0 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:52.688847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416b301b-d659-4224-be41-86002a79eeb0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:52.689329 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416b301b-d659-4224-be41-86002a79eeb0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:52.691564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416b301b-d659-4224-be41-86002a79eeb0 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:52.700199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 39/75] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:25:52 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:52.700986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416b301b-d659-4224-be41-86002a79eeb0 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:52.702125 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-90d077b0-53de-4210-a1e9-c6eafdb4864c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:52.703556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-90d077b0-53de-4210-a1e9-c6eafdb4864c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:52.771527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 37/76] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:52.772292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d077b0-53de-4210-a1e9-c6eafdb4864c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:52.779408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:52.801704 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416b301b-d659-4224-be41-86002a79eeb0 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:52.836550 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe58f070-1661-4a5a-9fbe-d53a466b8a81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:52.837180 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe58f070-1661-4a5a-9fbe-d53a466b8a81 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:52.837469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe58f070-1661-4a5a-9fbe-d53a466b8a81 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:52.837749 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe58f070-1661-4a5a-9fbe-d53a466b8a81 None admin] PATH_INFO: `/v3/users/18ccdd66160a423494d2172fb012e524` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:52.844009 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:52.851895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe58f070-1661-4a5a-9fbe-d53a466b8a81 None admin] RBAC: Authorizing `identity:get_user(user_id=18ccdd66160a423494d2172fb012e524)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:52.853362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe58f070-1661-4a5a-9fbe-d53a466b8a81 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:52.855030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 40/77] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:25:52 2022] GET /identity/v3/users/18ccdd66160a423494d2172fb012e524 => generated 275 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:52.855803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe58f070-1661-4a5a-9fbe-d53a466b8a81 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:52.899048 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d077b0-53de-4210-a1e9-c6eafdb4864c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:52.937783 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd88bd5e-a7bc-4fae-8fde-55bddc861b14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:52.938766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd88bd5e-a7bc-4fae-8fde-55bddc861b14 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:52.939057 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd88bd5e-a7bc-4fae-8fde-55bddc861b14 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:52.939534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd88bd5e-a7bc-4fae-8fde-55bddc861b14 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:52.950164 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd88bd5e-a7bc-4fae-8fde-55bddc861b14 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-bd88bd5e-a7bc-4fae-8fde-55bddc861b14 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:25:52.955296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:52.956147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:25:52.956936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 38/78] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:25:52 2022] GET /identity/v3/roles/member => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:25:52.957592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd88bd5e-a7bc-4fae-8fde-55bddc861b14 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:52.966579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe58f070-1661-4a5a-9fbe-d53a466b8a81 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:53.002888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ee9f2e-51d1-42bb-85f0-84d7ecb5e5ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:53.003505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ee9f2e-51d1-42bb-85f0-84d7ecb5e5ac None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:53.003786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ee9f2e-51d1-42bb-85f0-84d7ecb5e5ac None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:53.004060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ee9f2e-51d1-42bb-85f0-84d7ecb5e5ac None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:53.005294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ee9f2e-51d1-42bb-85f0-84d7ecb5e5ac None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:53.005716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ee9f2e-51d1-42bb-85f0-84d7ecb5e5ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:53.007503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ee9f2e-51d1-42bb-85f0-84d7ecb5e5ac None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:53.010208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:53.022024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 41/79] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:25:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:53.022024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ee9f2e-51d1-42bb-85f0-84d7ecb5e5ac None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:53.072088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd88bd5e-a7bc-4fae-8fde-55bddc861b14 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:53.086231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:53.105903 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9a5147-b7b4-467b-8a25-53724fd10682 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:53.106568 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9a5147-b7b4-467b-8a25-53724fd10682 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:53.106854 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9a5147-b7b4-467b-8a25-53724fd10682 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:53.107128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9a5147-b7b4-467b-8a25-53724fd10682 None admin] PATH_INFO: `/v3/users/18ccdd66160a423494d2172fb012e524` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:53.108289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9a5147-b7b4-467b-8a25-53724fd10682 None admin] RBAC: Authorizing `identity:get_user(user_id=18ccdd66160a423494d2172fb012e524)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:53.109764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9a5147-b7b4-467b-8a25-53724fd10682 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:53.111362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 39/80] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:25:52 2022] GET /identity/v3/users/18ccdd66160a423494d2172fb012e524 => generated 275 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:53.112128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9a5147-b7b4-467b-8a25-53724fd10682 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:53.128364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ee9f2e-51d1-42bb-85f0-84d7ecb5e5ac None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:53.162115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7771a5-6e01-407c-9fed-fb9260109104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:53.162761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7771a5-6e01-407c-9fed-fb9260109104 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:53.163042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7771a5-6e01-407c-9fed-fb9260109104 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:53.163316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7771a5-6e01-407c-9fed-fb9260109104 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:53.213675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9a5147-b7b4-467b-8a25-53724fd10682 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:53.227710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:53.231375 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7771a5-6e01-407c-9fed-fb9260109104 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:53.232854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7771a5-6e01-407c-9fed-fb9260109104 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:53.247188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3beeff3-571c-41bd-85ad-a63c7f89ec30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:53.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3beeff3-571c-41bd-85ad-a63c7f89ec30 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:53.248075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3beeff3-571c-41bd-85ad-a63c7f89ec30 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:53.248376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3beeff3-571c-41bd-85ad-a63c7f89ec30 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:53.249047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3beeff3-571c-41bd-85ad-a63c7f89ec30 None admin] RBAC: Adding query filter params (name=member) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:53.249345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3beeff3-571c-41bd-85ad-a63c7f89ec30 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:53.250742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3beeff3-571c-41bd-85ad-a63c7f89ec30 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:53.259855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 40/81] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:25:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:53.260422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3beeff3-571c-41bd-85ad-a63c7f89ec30 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:53.304423 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:53.336079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 42/82] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:25:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:25:53.336932 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7771a5-6e01-407c-9fed-fb9260109104 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:53.364096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3beeff3-571c-41bd-85ad-a63c7f89ec30 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:53.397288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d051517d-10cf-48ed-ab3b-ce8f4c7ad9cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:53.397911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d051517d-10cf-48ed-ab3b-ce8f4c7ad9cd None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:53.398209 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d051517d-10cf-48ed-ab3b-ce8f4c7ad9cd None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:53.398489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d051517d-10cf-48ed-ab3b-ce8f4c7ad9cd None admin] PATH_INFO: `/v3/users/c73de3f5e3b94a3182c45d2476903a00` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:53.399609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d051517d-10cf-48ed-ab3b-ce8f4c7ad9cd None admin] RBAC: Authorizing `identity:get_user(user_id=c73de3f5e3b94a3182c45d2476903a00)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:53.401078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d051517d-10cf-48ed-ab3b-ce8f4c7ad9cd None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:53.402676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 41/83] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:25:53 2022] GET /identity/v3/users/c73de3f5e3b94a3182c45d2476903a00 => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:53.403458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d051517d-10cf-48ed-ab3b-ce8f4c7ad9cd None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:53.435672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:53.439203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7771a5-6e01-407c-9fed-fb9260109104 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:53.475104 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54185d09-2c43-4ee3-aa60-62e893825286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:53.475849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54185d09-2c43-4ee3-aa60-62e893825286 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:53.476134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54185d09-2c43-4ee3-aa60-62e893825286 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:53.476409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54185d09-2c43-4ee3-aa60-62e893825286 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:53.477603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54185d09-2c43-4ee3-aa60-62e893825286 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:53.479169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54185d09-2c43-4ee3-aa60-62e893825286 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:53.480732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 43/84] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:25:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:53.485987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54185d09-2c43-4ee3-aa60-62e893825286 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:53.512032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d051517d-10cf-48ed-ab3b-ce8f4c7ad9cd None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:53.525064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:53.546179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad53ab6a-1fb2-4d0d-a270-fbc642aa1bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:53.546796 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad53ab6a-1fb2-4d0d-a270-fbc642aa1bd7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:53.547086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad53ab6a-1fb2-4d0d-a270-fbc642aa1bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:53.547364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad53ab6a-1fb2-4d0d-a270-fbc642aa1bd7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:53.585502 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54185d09-2c43-4ee3-aa60-62e893825286 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:53.615160 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad53ab6a-1fb2-4d0d-a270-fbc642aa1bd7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:53.616802 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad53ab6a-1fb2-4d0d-a270-fbc642aa1bd7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:53.618848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e595421-865f-42c0-b7bf-7854c2f79829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:53.619481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e595421-865f-42c0-b7bf-7854c2f79829 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:53.619763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e595421-865f-42c0-b7bf-7854c2f79829 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:53.620037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e595421-865f-42c0-b7bf-7854c2f79829 None admin] PATH_INFO: `/v3/users/18ccdd66160a423494d2172fb012e524` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:53.621244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e595421-865f-42c0-b7bf-7854c2f79829 None admin] RBAC: Authorizing `identity:get_user(user_id=18ccdd66160a423494d2172fb012e524)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:53.622730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e595421-865f-42c0-b7bf-7854c2f79829 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:53.624305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 44/85] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:25:53 2022] GET /identity/v3/users/18ccdd66160a423494d2172fb012e524 => generated 275 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:53.625066 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e595421-865f-42c0-b7bf-7854c2f79829 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:53.662552 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:53.718014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 42/86] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:25:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:25:53.718979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad53ab6a-1fb2-4d0d-a270-fbc642aa1bd7 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:53.727522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e595421-865f-42c0-b7bf-7854c2f79829 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:53.733672 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:53.762463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081cc224-0b0d-486a-8e03-0617efc6e712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:53.763097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081cc224-0b0d-486a-8e03-0617efc6e712 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:53.763379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081cc224-0b0d-486a-8e03-0617efc6e712 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:53.763653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081cc224-0b0d-486a-8e03-0617efc6e712 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:53.764809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081cc224-0b0d-486a-8e03-0617efc6e712 None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:53.766307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081cc224-0b0d-486a-8e03-0617efc6e712 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:53.767945 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 45/87] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:25:53 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:53.768776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081cc224-0b0d-486a-8e03-0617efc6e712 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:53.825236 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad53ab6a-1fb2-4d0d-a270-fbc642aa1bd7 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:53.862460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c361da-e3f6-49c4-8401-a09d95c485e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:53.863073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c361da-e3f6-49c4-8401-a09d95c485e0 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:53.863365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c361da-e3f6-49c4-8401-a09d95c485e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:53.864291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c361da-e3f6-49c4-8401-a09d95c485e0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:53.867693 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081cc224-0b0d-486a-8e03-0617efc6e712 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:53.869193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:53.901812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74a3e7a-0fd0-471c-8e43-94f7063f5374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:53.902499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74a3e7a-0fd0-471c-8e43-94f7063f5374 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:53.903064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74a3e7a-0fd0-471c-8e43-94f7063f5374 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:53.903568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74a3e7a-0fd0-471c-8e43-94f7063f5374 None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:53.904603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74a3e7a-0fd0-471c-8e43-94f7063f5374 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:53.906063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74a3e7a-0fd0-471c-8e43-94f7063f5374 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:53.932925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c361da-e3f6-49c4-8401-a09d95c485e0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:53.934461 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c361da-e3f6-49c4-8401-a09d95c485e0 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:53.947797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:53.989857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 46/88] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:25:53 2022] POST /identity/v3/users => generated 289 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:25:53.991103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74a3e7a-0fd0-471c-8e43-94f7063f5374 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:54.039776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 43/89] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:25:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:25:54.041005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c361da-e3f6-49c4-8401-a09d95c485e0 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:54.107597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74a3e7a-0fd0-471c-8e43-94f7063f5374 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:54.145512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214fc700-936b-4385-bc89-2fb36a7e80ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:54.146171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214fc700-936b-4385-bc89-2fb36a7e80ea None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:54.146480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214fc700-936b-4385-bc89-2fb36a7e80ea None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:54.148033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214fc700-936b-4385-bc89-2fb36a7e80ea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:54.166419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c361da-e3f6-49c4-8401-a09d95c485e0 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:54.181640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:54.182339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:54.206867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780bf562-0762-4115-a55d-4bb22084c8c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:54.208092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780bf562-0762-4115-a55d-4bb22084c8c0 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:54.208326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780bf562-0762-4115-a55d-4bb22084c8c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:54.208581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780bf562-0762-4115-a55d-4bb22084c8c0 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:54.209747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780bf562-0762-4115-a55d-4bb22084c8c0 None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:54.211279 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780bf562-0762-4115-a55d-4bb22084c8c0 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:54.213016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 44/90] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:25:54 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:54.213739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780bf562-0762-4115-a55d-4bb22084c8c0 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:54.233124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214fc700-936b-4385-bc89-2fb36a7e80ea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:54.235131 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214fc700-936b-4385-bc89-2fb36a7e80ea None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:54.353794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 47/91] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:25:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:25:54.354913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214fc700-936b-4385-bc89-2fb36a7e80ea None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:54.385760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780bf562-0762-4115-a55d-4bb22084c8c0 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:54.429748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1e7150-2c0e-4cc0-b5d7-eda39482f4a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:54.430402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1e7150-2c0e-4cc0-b5d7-eda39482f4a2 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:54.430745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1e7150-2c0e-4cc0-b5d7-eda39482f4a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:54.431025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1e7150-2c0e-4cc0-b5d7-eda39482f4a2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:54.431764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1e7150-2c0e-4cc0-b5d7-eda39482f4a2 None admin] RBAC: Adding query filter params (role.id=9912acfb698e41ebbce6a5147fafe896, scope.project.id=db44137c64a347cfb350dad24d36a3db, user.id=18ccdd66160a423494d2172fb012e524) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:54.432083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1e7150-2c0e-4cc0-b5d7-eda39482f4a2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:54.433599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1e7150-2c0e-4cc0-b5d7-eda39482f4a2 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:54.439697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 45/92] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:25:54 2022] GET /identity/v3/role_assignments?user.id=18ccdd66160a423494d2172fb012e524&scope.project.id=db44137c64a347cfb350dad24d36a3db&role.id=9912acfb698e41ebbce6a5147fafe896 => generated 259 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:54.446200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:54.446480 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1e7150-2c0e-4cc0-b5d7-eda39482f4a2 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:54.528080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:54.543077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214fc700-936b-4385-bc89-2fb36a7e80ea None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:54.609442 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d450e209-9740-4c9d-b922-a6778686cfd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:54.610092 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d450e209-9740-4c9d-b922-a6778686cfd6 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:54.610397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d450e209-9740-4c9d-b922-a6778686cfd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:54.610676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d450e209-9740-4c9d-b922-a6778686cfd6 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:54.615355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d450e209-9740-4c9d-b922-a6778686cfd6 None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:54.616855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d450e209-9740-4c9d-b922-a6778686cfd6 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:54.618618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 48/93] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:25:54 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:54.619409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d450e209-9740-4c9d-b922-a6778686cfd6 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:54.638758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1e7150-2c0e-4cc0-b5d7-eda39482f4a2 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:54.689610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cccda3-e268-4edb-9154-2c3ab94cabda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:54.690588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cccda3-e268-4edb-9154-2c3ab94cabda None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:54.691019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cccda3-e268-4edb-9154-2c3ab94cabda None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:54.691743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cccda3-e268-4edb-9154-2c3ab94cabda None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:54.693534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10cccda3-e268-4edb-9154-2c3ab94cabda None admin] RBAC: Adding query filter params (role.id=d02a8e6c49b74aaa9403f8bd535b90bf, scope.project.id=db44137c64a347cfb350dad24d36a3db, user.id=18ccdd66160a423494d2172fb012e524) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:54.694004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10cccda3-e268-4edb-9154-2c3ab94cabda None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:54.695903 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10cccda3-e268-4edb-9154-2c3ab94cabda None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:54.703666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 46/94] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:25:54 2022] GET /identity/v3/role_assignments?user.id=18ccdd66160a423494d2172fb012e524&scope.project.id=db44137c64a347cfb350dad24d36a3db&role.id=d02a8e6c49b74aaa9403f8bd535b90bf => generated 259 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:54.704878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cccda3-e268-4edb-9154-2c3ab94cabda None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:54.778965 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d450e209-9740-4c9d-b922-a6778686cfd6 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:54.807033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:54.816554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df75260-bf23-45a0-841d-0c35cae16090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:54.817769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df75260-bf23-45a0-841d-0c35cae16090 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:54.818258 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df75260-bf23-45a0-841d-0c35cae16090 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:54.818692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df75260-bf23-45a0-841d-0c35cae16090 None admin] PATH_INFO: `/v3/projects/86c05145fc0e4ca79b274eac020a148c` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:54.828137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df75260-bf23-45a0-841d-0c35cae16090 None admin] RBAC: Authorizing `identity:get_project(project_id=86c05145fc0e4ca79b274eac020a148c)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:54.830642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df75260-bf23-45a0-841d-0c35cae16090 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:54.835293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 49/95] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:25:54 2022] GET /identity/v3/projects/86c05145fc0e4ca79b274eac020a148c => generated 312 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:54.844051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df75260-bf23-45a0-841d-0c35cae16090 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:54.946129 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cccda3-e268-4edb-9154-2c3ab94cabda None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:54.966900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:55.010404 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df75260-bf23-45a0-841d-0c35cae16090 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:55.054124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e2b075-218e-4a23-8f58-4baff61f2647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:55.054822 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e2b075-218e-4a23-8f58-4baff61f2647 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:55.055114 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e2b075-218e-4a23-8f58-4baff61f2647 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:55.055393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e2b075-218e-4a23-8f58-4baff61f2647 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:55.055663 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1becfdf7-a2a9-439d-913b-669cf3766638 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:55.056130 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e2b075-218e-4a23-8f58-4baff61f2647 None admin] RBAC: Adding query filter params (role.id=9912acfb698e41ebbce6a5147fafe896, scope.project.id=86c05145fc0e4ca79b274eac020a148c, user.id=18ccdd66160a423494d2172fb012e524) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:55.056437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e2b075-218e-4a23-8f58-4baff61f2647 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:55.056900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1becfdf7-a2a9-439d-913b-669cf3766638 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:55.057349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1becfdf7-a2a9-439d-913b-669cf3766638 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:55.057801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1becfdf7-a2a9-439d-913b-669cf3766638 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:55.057994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e2b075-218e-4a23-8f58-4baff61f2647 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:55.058894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1becfdf7-a2a9-439d-913b-669cf3766638 None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=db44137c64a347cfb350dad24d36a3db, user.id=c73de3f5e3b94a3182c45d2476903a00) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:55.059376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1becfdf7-a2a9-439d-913b-669cf3766638 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:55.060986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1becfdf7-a2a9-439d-913b-669cf3766638 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:55.071107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 50/96] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:25:54 2022] GET /identity/v3/role_assignments?user.id=18ccdd66160a423494d2172fb012e524&scope.project.id=86c05145fc0e4ca79b274eac020a148c&role.id=9912acfb698e41ebbce6a5147fafe896 => generated 259 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:55.080378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 47/97] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:25:54 2022] GET /identity/v3/role_assignments?user.id=c73de3f5e3b94a3182c45d2476903a00&scope.project.id=db44137c64a347cfb350dad24d36a3db&role.id=43ca25467bc6484da3a50508efb79230 => generated 259 bytes in 376 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:56.464682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dfdcce-f9c1-4490-9806-29b560438511 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:56.465665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dfdcce-f9c1-4490-9806-29b560438511 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:56.466200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dfdcce-f9c1-4490-9806-29b560438511 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:56.467755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 51/98] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:56.473182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6cf0d0-e580-479c-a8ca-e0a4ac2e2628 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:56.473570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6cf0d0-e580-479c-a8ca-e0a4ac2e2628 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:56.473830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6cf0d0-e580-479c-a8ca-e0a4ac2e2628 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:56.502121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-ca6cf0d0-e580-479c-a8ca-e0a4ac2e2628 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:56.503365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ca6cf0d0-e580-479c-a8ca-e0a4ac2e2628 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:56.591739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 48/99] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:56.602096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4872d8d-4fdd-46c3-98eb-28cbb0370f35 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:56.602409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4872d8d-4fdd-46c3-98eb-28cbb0370f35 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:56.602682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4872d8d-4fdd-46c3-98eb-28cbb0370f35 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:56.604020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 52/100] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:56.609107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c12e2e-f3e4-4f1f-8d26-afc1076c2794 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:56.609397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c12e2e-f3e4-4f1f-8d26-afc1076c2794 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:56.609669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c12e2e-f3e4-4f1f-8d26-afc1076c2794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:56.639125 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-32c12e2e-f3e4-4f1f-8d26-afc1076c2794 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:56.640320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-32c12e2e-f3e4-4f1f-8d26-afc1076c2794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:56.724734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 49/101] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:56.965313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4872d8d-4fdd-46c3-98eb-28cbb0370f35 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:57.044246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:57.099206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4872d8d-4fdd-46c3-98eb-28cbb0370f35 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:57.137780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639c81db-d8cb-4ab9-b475-c0a0aee4f23e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:57.138453 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639c81db-d8cb-4ab9-b475-c0a0aee4f23e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:57.139269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639c81db-d8cb-4ab9-b475-c0a0aee4f23e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:57.139553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639c81db-d8cb-4ab9-b475-c0a0aee4f23e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:57.156000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639c81db-d8cb-4ab9-b475-c0a0aee4f23e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-639c81db-d8cb-4ab9-b475-c0a0aee4f23e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:25:57.161528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:57.162379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:25:57.163125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 53/102] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:25:56 2022] GET /identity/v3/roles/reader => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:25:57.168486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c12e2e-f3e4-4f1f-8d26-afc1076c2794 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:57.226673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:57.261696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639c81db-d8cb-4ab9-b475-c0a0aee4f23e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:57.323307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c12e2e-f3e4-4f1f-8d26-afc1076c2794 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:57.388377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d8dfff-7a1a-4997-888a-c21135753ea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:57.389083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d8dfff-7a1a-4997-888a-c21135753ea5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:57.389374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d8dfff-7a1a-4997-888a-c21135753ea5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:57.389650 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d8dfff-7a1a-4997-888a-c21135753ea5 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:57.390459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d8dfff-7a1a-4997-888a-c21135753ea5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:57.390760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d8dfff-7a1a-4997-888a-c21135753ea5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:57.392137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d8dfff-7a1a-4997-888a-c21135753ea5 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:57.422086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 50/103] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:25:57 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:57.427911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d8dfff-7a1a-4997-888a-c21135753ea5 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:57.558318 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639c81db-d8cb-4ab9-b475-c0a0aee4f23e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:57.596753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:57.606291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d8dfff-7a1a-4997-888a-c21135753ea5 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:57.607896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:57.649772 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caeb682-90e5-4cb0-ba0c-82d72b148ef6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:57.650457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caeb682-90e5-4cb0-ba0c-82d72b148ef6 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:57.650743 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caeb682-90e5-4cb0-ba0c-82d72b148ef6 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:57.651045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caeb682-90e5-4cb0-ba0c-82d72b148ef6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:57.653652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7caeb682-90e5-4cb0-ba0c-82d72b148ef6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:57.655163 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7caeb682-90e5-4cb0-ba0c-82d72b148ef6 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:57.657169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 54/104] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:25:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 396 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:25:57.658364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9871ed-c09f-46a7-9411-ea96a48ce91b None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:57.658670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9871ed-c09f-46a7-9411-ea96a48ce91b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:57.658896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b42998-1ed9-4d0a-9981-5ced3f471aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:57.659090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9871ed-c09f-46a7-9411-ea96a48ce91b None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:57.659548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b42998-1ed9-4d0a-9981-5ced3f471aaf None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:57.659835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b42998-1ed9-4d0a-9981-5ced3f471aaf None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:57.660212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b42998-1ed9-4d0a-9981-5ced3f471aaf None admin] PATH_INFO: `/v3/users/6a30a0b87e6d4e788cbc5f35c9605f6b` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:57.660521 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 55/105] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:57.666053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9871ed-c09f-46a7-9411-ea96a48ce91b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:57.692992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b42998-1ed9-4d0a-9981-5ced3f471aaf None admin] RBAC: Authorizing `identity:get_user(user_id=6a30a0b87e6d4e788cbc5f35c9605f6b)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:57.694493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b42998-1ed9-4d0a-9981-5ced3f471aaf None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:57.696190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 51/106] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:25:57 2022] GET /identity/v3/users/6a30a0b87e6d4e788cbc5f35c9605f6b => generated 289 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:25:57.697435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037f7687-2aa1-4d70-93d6-c25782f2afb9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:57.697712 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037f7687-2aa1-4d70-93d6-c25782f2afb9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:57.697909 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037f7687-2aa1-4d70-93d6-c25782f2afb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:57.736660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-037f7687-2aa1-4d70-93d6-c25782f2afb9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:57.737942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-037f7687-2aa1-4d70-93d6-c25782f2afb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:57.820801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 52/107] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:57.821683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037f7687-2aa1-4d70-93d6-c25782f2afb9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:57.874360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9871ed-c09f-46a7-9411-ea96a48ce91b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:57.906679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:57.925107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037f7687-2aa1-4d70-93d6-c25782f2afb9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:57.937133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c144b5-d88a-4861-886b-820fbba3c6f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:57.937753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c144b5-d88a-4861-886b-820fbba3c6f7 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:57.938094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c144b5-d88a-4861-886b-820fbba3c6f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:57.938638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c144b5-d88a-4861-886b-820fbba3c6f7 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:57.939298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c144b5-d88a-4861-886b-820fbba3c6f7 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:57.940770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c144b5-d88a-4861-886b-820fbba3c6f7 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:57.982865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fe4d89-a03f-4afd-b1bf-bca3c55bdc89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:57.983473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fe4d89-a03f-4afd-b1bf-bca3c55bdc89 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:57.983756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fe4d89-a03f-4afd-b1bf-bca3c55bdc89 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:57.984026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fe4d89-a03f-4afd-b1bf-bca3c55bdc89 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:58.053852 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fe4d89-a03f-4afd-b1bf-bca3c55bdc89 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:58.055346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fe4d89-a03f-4afd-b1bf-bca3c55bdc89 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:58.071828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 56/108] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:25:57 2022] POST /identity/v3/projects => generated 302 bytes in 406 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:25:58.071828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e51a58-ef94-4c7e-873b-460de57e9f8d None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:58.071828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e51a58-ef94-4c7e-873b-460de57e9f8d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:58.071828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e51a58-ef94-4c7e-873b-460de57e9f8d None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:58.071828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 57/109] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:58.071828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20576325-1914-4700-aefa-c65a7863a1a1 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:58.071828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20576325-1914-4700-aefa-c65a7863a1a1 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:58.080052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20576325-1914-4700-aefa-c65a7863a1a1 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:58.083049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 58/110] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:58 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:58.084637 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d59c13-af9e-4e2d-9606-1c260b30765f None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:58.088908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d59c13-af9e-4e2d-9606-1c260b30765f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:58.088908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d59c13-af9e-4e2d-9606-1c260b30765f None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:58.088908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 59/111] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:25:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:25:58.088908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801c0ffb-0ad2-4484-a5c2-ed17f15818c8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:58.088908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801c0ffb-0ad2-4484-a5c2-ed17f15818c8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:58.088908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801c0ffb-0ad2-4484-a5c2-ed17f15818c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:58.113400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-801c0ffb-0ad2-4484-a5c2-ed17f15818c8 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:58.115968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-801c0ffb-0ad2-4484-a5c2-ed17f15818c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:58.150312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:58.178553 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:58.188812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 53/112] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:25:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:25:58.195536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6983bb87-f568-403e-beb1-df67ca68b7b3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:58.195818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6983bb87-f568-403e-beb1-df67ca68b7b3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:58.196097 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6983bb87-f568-403e-beb1-df67ca68b7b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:58.200827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 60/113] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:58.202389 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b3dd88-2969-4637-bbe6-93c8e1d33eae None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:58.202683 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b3dd88-2969-4637-bbe6-93c8e1d33eae None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:58.202963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b3dd88-2969-4637-bbe6-93c8e1d33eae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:58.224732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-6983bb87-f568-403e-beb1-df67ca68b7b3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:58.225700 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-99b3dd88-2969-4637-bbe6-93c8e1d33eae None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:25:58.226058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6983bb87-f568-403e-beb1-df67ca68b7b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:58.227302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-99b3dd88-2969-4637-bbe6-93c8e1d33eae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:25:58.308608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 54/114] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:58.309332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 61/115] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:25:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:25:58.309518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6983bb87-f568-403e-beb1-df67ca68b7b3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:58.310618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b3dd88-2969-4637-bbe6-93c8e1d33eae None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:58.418639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:58.422635 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b3dd88-2969-4637-bbe6-93c8e1d33eae None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:58.472065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad90d28-8cba-4c4e-9253-5a83e52812e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:58.473317 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad90d28-8cba-4c4e-9253-5a83e52812e8 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:58.474026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad90d28-8cba-4c4e-9253-5a83e52812e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:58.474554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad90d28-8cba-4c4e-9253-5a83e52812e8 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:58.475905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad90d28-8cba-4c4e-9253-5a83e52812e8 None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:58.477694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad90d28-8cba-4c4e-9253-5a83e52812e8 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:58.479563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 62/116] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:25:58 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:58.486741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad90d28-8cba-4c4e-9253-5a83e52812e8 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:58.560336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6983bb87-f568-403e-beb1-df67ca68b7b3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:58.587728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:58.632787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9969a25-8b03-49d6-8f4b-6a10f7be5964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:58.634246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9969a25-8b03-49d6-8f4b-6a10f7be5964 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:58.634718 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9969a25-8b03-49d6-8f4b-6a10f7be5964 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:58.635200 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9969a25-8b03-49d6-8f4b-6a10f7be5964 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:58.650572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9969a25-8b03-49d6-8f4b-6a10f7be5964 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:25:58.651946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad90d28-8cba-4c4e-9253-5a83e52812e8 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-b9969a25-8b03-49d6-8f4b-6a10f7be5964 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:25:58.656837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:58.657738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:25:58.658513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 55/117] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:25:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 349 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:25:58.659166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9969a25-8b03-49d6-8f4b-6a10f7be5964 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:58.700920 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd35444-d7cf-47cf-8417-65561348afc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:58.701880 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd35444-d7cf-47cf-8417-65561348afc1 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:58.702829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd35444-d7cf-47cf-8417-65561348afc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:58.703389 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd35444-d7cf-47cf-8417-65561348afc1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:58.704381 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd35444-d7cf-47cf-8417-65561348afc1 None admin] RBAC: Adding query filter params (role.id=74e1fd5acabf47a7b788b2dc78d1fb3e, scope.project.id=db44137c64a347cfb350dad24d36a3db, user.id=6a30a0b87e6d4e788cbc5f35c9605f6b) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:58.704899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd35444-d7cf-47cf-8417-65561348afc1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:58.707660 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd35444-d7cf-47cf-8417-65561348afc1 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:58.715727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 63/118] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:25:58 2022] GET /identity/v3/role_assignments?user.id=6a30a0b87e6d4e788cbc5f35c9605f6b&scope.project.id=db44137c64a347cfb350dad24d36a3db&role.id=74e1fd5acabf47a7b788b2dc78d1fb3e => generated 259 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:58.718478 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd35444-d7cf-47cf-8417-65561348afc1 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:58.772708 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:58.787900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9969a25-8b03-49d6-8f4b-6a10f7be5964 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:58.849870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c040ade6-de05-4fd0-9254-fd5f11d4de6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:58.850562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c040ade6-de05-4fd0-9254-fd5f11d4de6f None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:58.850838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c040ade6-de05-4fd0-9254-fd5f11d4de6f None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:58.851106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c040ade6-de05-4fd0-9254-fd5f11d4de6f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:58.863065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c040ade6-de05-4fd0-9254-fd5f11d4de6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:25:58.864825 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd35444-d7cf-47cf-8417-65561348afc1 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-c040ade6-de05-4fd0-9254-fd5f11d4de6f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:25:58.869191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:58.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 56/119] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:25:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:25:58.876275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c040ade6-de05-4fd0-9254-fd5f11d4de6f None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:58.912838 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a616510c-c047-4d4a-8635-e5f2ebad53f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:58.913484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a616510c-c047-4d4a-8635-e5f2ebad53f9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:58.913820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a616510c-c047-4d4a-8635-e5f2ebad53f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:58.914104 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a616510c-c047-4d4a-8635-e5f2ebad53f9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:58.918381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:58.925501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a616510c-c047-4d4a-8635-e5f2ebad53f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-a616510c-c047-4d4a-8635-e5f2ebad53f9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:25:58.932926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:58.934202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:25:58.934978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 64/120] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:25:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:25:58.935693 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a616510c-c047-4d4a-8635-e5f2ebad53f9 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:59.018204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c040ade6-de05-4fd0-9254-fd5f11d4de6f None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:59.049004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:59.056380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01647d5a-27de-4a18-816f-c2b798ed9614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:59.061694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01647d5a-27de-4a18-816f-c2b798ed9614 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:59.061694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01647d5a-27de-4a18-816f-c2b798ed9614 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:59.061694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01647d5a-27de-4a18-816f-c2b798ed9614 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:59.061694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01647d5a-27de-4a18-816f-c2b798ed9614 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:59.061694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01647d5a-27de-4a18-816f-c2b798ed9614 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:59.061694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01647d5a-27de-4a18-816f-c2b798ed9614 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:59.062138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a616510c-c047-4d4a-8635-e5f2ebad53f9 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:59.071176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 57/121] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:25:58 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:59.072310 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01647d5a-27de-4a18-816f-c2b798ed9614 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:59.106715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d4d0c7-3603-4c4d-8f14-c8ea74ebc3b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:59.107863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d4d0c7-3603-4c4d-8f14-c8ea74ebc3b8 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:59.108239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d4d0c7-3603-4c4d-8f14-c8ea74ebc3b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:59.108975 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d4d0c7-3603-4c4d-8f14-c8ea74ebc3b8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:59.122656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d4d0c7-3603-4c4d-8f14-c8ea74ebc3b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-98d4d0c7-3603-4c4d-8f14-c8ea74ebc3b8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:25:59.130078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:25:59.131428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:25:59.132400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:25:59.132400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:25:59.132400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:25:59.132400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:25:59.132400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:25:59.132400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:25:59.132400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:25:59.132400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:25:59.132400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:25:59.132400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:25:59.132400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:25:59.132400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:25:59.134215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 65/122] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:25:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:25:59.135279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d4d0c7-3603-4c4d-8f14-c8ea74ebc3b8 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:59.201332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:59.209012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01647d5a-27de-4a18-816f-c2b798ed9614 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:59.248317 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d5da6c-b02c-4e58-a130-c31489713938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:59.248934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d5da6c-b02c-4e58-a130-c31489713938 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:59.249214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d5da6c-b02c-4e58-a130-c31489713938 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:59.249487 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d5da6c-b02c-4e58-a130-c31489713938 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:59.250300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d5da6c-b02c-4e58-a130-c31489713938 None admin] RBAC: Adding query filter params (name=member) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:59.250605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d5da6c-b02c-4e58-a130-c31489713938 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:59.252166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d5da6c-b02c-4e58-a130-c31489713938 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:59.260851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d4d0c7-3603-4c4d-8f14-c8ea74ebc3b8 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:59.265980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 58/123] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:25:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:59.265980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d5da6c-b02c-4e58-a130-c31489713938 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:59.309072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9a789c-309f-4809-9530-3cb6061e463a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:59.310207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9a789c-309f-4809-9530-3cb6061e463a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:59.310691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9a789c-309f-4809-9530-3cb6061e463a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:59.311179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9a789c-309f-4809-9530-3cb6061e463a None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:59.312164 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9a789c-309f-4809-9530-3cb6061e463a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:59.312709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9a789c-309f-4809-9530-3cb6061e463a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:59.314445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9a789c-309f-4809-9530-3cb6061e463a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:59.324009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 66/124] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:25:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:59.325200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9a789c-309f-4809-9530-3cb6061e463a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:59.342355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:59.379081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d5da6c-b02c-4e58-a130-c31489713938 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:59.424734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9386dae-f4d1-4f9a-ae4c-034b375de100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:59.425524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9386dae-f4d1-4f9a-ae4c-034b375de100 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:59.425631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9386dae-f4d1-4f9a-ae4c-034b375de100 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:59.426004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9386dae-f4d1-4f9a-ae4c-034b375de100 None admin] PATH_INFO: `/v3/users/18ccdd66160a423494d2172fb012e524` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:59.427276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9386dae-f4d1-4f9a-ae4c-034b375de100 None admin] RBAC: Authorizing `identity:get_user(user_id=18ccdd66160a423494d2172fb012e524)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:59.428708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9386dae-f4d1-4f9a-ae4c-034b375de100 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:59.430251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 59/125] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:25:59 2022] GET /identity/v3/users/18ccdd66160a423494d2172fb012e524 => generated 275 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:59.431026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9386dae-f4d1-4f9a-ae4c-034b375de100 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:59.446921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9a789c-309f-4809-9530-3cb6061e463a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:59.461379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:59.485394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c5df19-5ab5-4f8a-9f24-d564c4009e6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:59.486333 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c5df19-5ab5-4f8a-9f24-d564c4009e6d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:59.486838 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c5df19-5ab5-4f8a-9f24-d564c4009e6d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:59.487342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c5df19-5ab5-4f8a-9f24-d564c4009e6d None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:59.488851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c5df19-5ab5-4f8a-9f24-d564c4009e6d None admin] RBAC: Adding query filter params (name=member) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:25:59.489357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c5df19-5ab5-4f8a-9f24-d564c4009e6d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:59.490960 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c5df19-5ab5-4f8a-9f24-d564c4009e6d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:59.506112 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 67/126] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:25:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:59.507398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c5df19-5ab5-4f8a-9f24-d564c4009e6d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:59.536977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9386dae-f4d1-4f9a-ae4c-034b375de100 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:59.572639 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f642be61-2a47-494b-8d2f-96d52ce6a8a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:59.573202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f642be61-2a47-494b-8d2f-96d52ce6a8a1 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:59.573483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f642be61-2a47-494b-8d2f-96d52ce6a8a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:59.573757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f642be61-2a47-494b-8d2f-96d52ce6a8a1 None admin] PATH_INFO: `/v3/users/18ccdd66160a423494d2172fb012e524` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:59.574921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f642be61-2a47-494b-8d2f-96d52ce6a8a1 None admin] RBAC: Authorizing `identity:get_user(user_id=18ccdd66160a423494d2172fb012e524)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:59.576355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f642be61-2a47-494b-8d2f-96d52ce6a8a1 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:59.577924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 60/127] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:25:59 2022] GET /identity/v3/users/18ccdd66160a423494d2172fb012e524 => generated 275 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:59.578761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f642be61-2a47-494b-8d2f-96d52ce6a8a1 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:59.597601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:59.627499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c5df19-5ab5-4f8a-9f24-d564c4009e6d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:59.669639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ec0467-a47c-4f6d-9aff-cabe50f113c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:59.671061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ec0467-a47c-4f6d-9aff-cabe50f113c0 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:59.671569 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ec0467-a47c-4f6d-9aff-cabe50f113c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:59.672142 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ec0467-a47c-4f6d-9aff-cabe50f113c0 None admin] PATH_INFO: `/v3/users/c73de3f5e3b94a3182c45d2476903a00` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:59.672953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:59.674940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ec0467-a47c-4f6d-9aff-cabe50f113c0 None admin] RBAC: Authorizing `identity:get_user(user_id=c73de3f5e3b94a3182c45d2476903a00)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:59.676897 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ec0467-a47c-4f6d-9aff-cabe50f113c0 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:59.678940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 68/128] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:25:59 2022] GET /identity/v3/users/c73de3f5e3b94a3182c45d2476903a00 => generated 247 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:59.679899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ec0467-a47c-4f6d-9aff-cabe50f113c0 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:59.681915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f642be61-2a47-494b-8d2f-96d52ce6a8a1 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:59.718639 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57402238-f113-4bf2-a320-029ccb13d14e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:59.719280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57402238-f113-4bf2-a320-029ccb13d14e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:59.719571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57402238-f113-4bf2-a320-029ccb13d14e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:59.719845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57402238-f113-4bf2-a320-029ccb13d14e None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:59.723443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57402238-f113-4bf2-a320-029ccb13d14e None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:59.725133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57402238-f113-4bf2-a320-029ccb13d14e None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:59.728234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 61/129] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:25:59 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:59.729294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57402238-f113-4bf2-a320-029ccb13d14e None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:59.801699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ec0467-a47c-4f6d-9aff-cabe50f113c0 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:59.846539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2495c5e-92aa-49af-ad45-a0cb49e30f1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:59.847527 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2495c5e-92aa-49af-ad45-a0cb49e30f1a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:59.848727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2495c5e-92aa-49af-ad45-a0cb49e30f1a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:59.849350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2495c5e-92aa-49af-ad45-a0cb49e30f1a None admin] PATH_INFO: `/v3/users/18ccdd66160a423494d2172fb012e524` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:59.850854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2495c5e-92aa-49af-ad45-a0cb49e30f1a None admin] RBAC: Authorizing `identity:get_user(user_id=18ccdd66160a423494d2172fb012e524)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:59.852478 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2495c5e-92aa-49af-ad45-a0cb49e30f1a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:59.854426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 69/130] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:25:59 2022] GET /identity/v3/users/18ccdd66160a423494d2172fb012e524 => generated 275 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:59.855361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2495c5e-92aa-49af-ad45-a0cb49e30f1a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:59.866395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57402238-f113-4bf2-a320-029ccb13d14e None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:25:59.872688 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:59.903226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f81cea-f66a-4a43-857d-499ea233b027 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:25:59.904116 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f81cea-f66a-4a43-857d-499ea233b027 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:25:59.904342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f81cea-f66a-4a43-857d-499ea233b027 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:25:59.904697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f81cea-f66a-4a43-857d-499ea233b027 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:25:59.905965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f81cea-f66a-4a43-857d-499ea233b027 None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:25:59.907525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f81cea-f66a-4a43-857d-499ea233b027 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:25:59.909241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 62/131] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:25:59 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:25:59.910128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f81cea-f66a-4a43-857d-499ea233b027 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:25:59.939942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:25:59.969318 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2495c5e-92aa-49af-ad45-a0cb49e30f1a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:00.007100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8714c4e-86b0-4a03-a6d7-74c6afeab6d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:00.008004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8714c4e-86b0-4a03-a6d7-74c6afeab6d9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:00.008454 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8714c4e-86b0-4a03-a6d7-74c6afeab6d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:00.008933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8714c4e-86b0-4a03-a6d7-74c6afeab6d9 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:00.010388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8714c4e-86b0-4a03-a6d7-74c6afeab6d9 None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:00.012335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8714c4e-86b0-4a03-a6d7-74c6afeab6d9 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:00.014169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 70/132] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:25:59 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:00.015088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8714c4e-86b0-4a03-a6d7-74c6afeab6d9 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:00.015676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f81cea-f66a-4a43-857d-499ea233b027 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:00.052821 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d43b50-076a-488a-bea7-e76d7e659cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:00.053644 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d43b50-076a-488a-bea7-e76d7e659cdf None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:00.054021 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d43b50-076a-488a-bea7-e76d7e659cdf None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:00.054402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d43b50-076a-488a-bea7-e76d7e659cdf None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db/users/18ccdd66160a423494d2172fb012e524/roles/d02a8e6c49b74aaa9403f8bd535b90bf` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:00.057189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d43b50-076a-488a-bea7-e76d7e659cdf None admin] RBAC: Authorizing `identity:create_grant(project_id=db44137c64a347cfb350dad24d36a3db, user_id=18ccdd66160a423494d2172fb012e524, role_id=d02a8e6c49b74aaa9403f8bd535b90bf)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:00.058835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d43b50-076a-488a-bea7-e76d7e659cdf None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:00.100732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:00.122918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8714c4e-86b0-4a03-a6d7-74c6afeab6d9 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:00.124045 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 63/133] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:25:59 2022] PUT /identity/v3/projects/db44137c64a347cfb350dad24d36a3db/users/18ccdd66160a423494d2172fb012e524/roles/d02a8e6c49b74aaa9403f8bd535b90bf => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:00.129918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d43b50-076a-488a-bea7-e76d7e659cdf None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:00.196242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fbbf7f-f434-49f0-a66d-358fe89880f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:00.198355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fbbf7f-f434-49f0-a66d-358fe89880f9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:00.198936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fbbf7f-f434-49f0-a66d-358fe89880f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:00.200048 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fbbf7f-f434-49f0-a66d-358fe89880f9 None admin] PATH_INFO: `/v3/projects/86c05145fc0e4ca79b274eac020a148c` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:00.202990 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fbbf7f-f434-49f0-a66d-358fe89880f9 None admin] RBAC: Authorizing `identity:get_project(project_id=86c05145fc0e4ca79b274eac020a148c)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:00.205481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fbbf7f-f434-49f0-a66d-358fe89880f9 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:00.212758 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 71/134] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:00 2022] GET /identity/v3/projects/86c05145fc0e4ca79b274eac020a148c => generated 312 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:00.213915 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fbbf7f-f434-49f0-a66d-358fe89880f9 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:00.318236 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:00.334239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d43b50-076a-488a-bea7-e76d7e659cdf None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:00.352572 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fbbf7f-f434-49f0-a66d-358fe89880f9 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:00.372208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd241d3-d4d2-4baf-b1f1-977d5aee881f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:00.373027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd241d3-d4d2-4baf-b1f1-977d5aee881f None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:00.373402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd241d3-d4d2-4baf-b1f1-977d5aee881f None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:00.373773 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd241d3-d4d2-4baf-b1f1-977d5aee881f None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db/users/18ccdd66160a423494d2172fb012e524/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:00.375834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd241d3-d4d2-4baf-b1f1-977d5aee881f None admin] RBAC: Authorizing `identity:create_grant(project_id=db44137c64a347cfb350dad24d36a3db, user_id=18ccdd66160a423494d2172fb012e524, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:00.377636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd241d3-d4d2-4baf-b1f1-977d5aee881f None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:00.394000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57945987-f609-427f-add6-ada390fdd520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:00.395418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57945987-f609-427f-add6-ada390fdd520 None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:00.395720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57945987-f609-427f-add6-ada390fdd520 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:00.395988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57945987-f609-427f-add6-ada390fdd520 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db/users/c73de3f5e3b94a3182c45d2476903a00/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:00.397924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57945987-f609-427f-add6-ada390fdd520 None admin] RBAC: Authorizing `identity:create_grant(project_id=db44137c64a347cfb350dad24d36a3db, user_id=c73de3f5e3b94a3182c45d2476903a00, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:00.399466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57945987-f609-427f-add6-ada390fdd520 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:00.470143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 64/135] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:26:00 2022] PUT /identity/v3/projects/db44137c64a347cfb350dad24d36a3db/users/18ccdd66160a423494d2172fb012e524/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:00.476912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd241d3-d4d2-4baf-b1f1-977d5aee881f None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:00.500713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 72/136] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:26:00 2022] PUT /identity/v3/projects/db44137c64a347cfb350dad24d36a3db/users/c73de3f5e3b94a3182c45d2476903a00/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:00.501970 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf73e2f-1cd6-4f6e-b2e5-4a43bf51fafd None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:00.502302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf73e2f-1cd6-4f6e-b2e5-4a43bf51fafd None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:00.502750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf73e2f-1cd6-4f6e-b2e5-4a43bf51fafd None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:00.505302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 73/137] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:00.511709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd54d49-2ffa-418a-8197-654b3ed68766 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:00.511993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd54d49-2ffa-418a-8197-654b3ed68766 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:00.512263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd54d49-2ffa-418a-8197-654b3ed68766 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:00.537735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:00.548221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-efd54d49-2ffa-418a-8197-654b3ed68766 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:00.550171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-efd54d49-2ffa-418a-8197-654b3ed68766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:00.671517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd241d3-d4d2-4baf-b1f1-977d5aee881f None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:00.711693 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 74/138] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:26:00.714355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37bf5c7-6361-4af4-b40b-4aa475db992b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:00.715120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37bf5c7-6361-4af4-b40b-4aa475db992b None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:00.715453 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37bf5c7-6361-4af4-b40b-4aa475db992b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:00.715768 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37bf5c7-6361-4af4-b40b-4aa475db992b None admin] PATH_INFO: `/v3/projects/86c05145fc0e4ca79b274eac020a148c/users/18ccdd66160a423494d2172fb012e524/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:00.717717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37bf5c7-6361-4af4-b40b-4aa475db992b None admin] RBAC: Authorizing `identity:create_grant(project_id=86c05145fc0e4ca79b274eac020a148c, user_id=18ccdd66160a423494d2172fb012e524, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:00.719204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37bf5c7-6361-4af4-b40b-4aa475db992b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:00.791056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 65/139] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:26:00 2022] PUT /identity/v3/projects/86c05145fc0e4ca79b274eac020a148c/users/18ccdd66160a423494d2172fb012e524/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:00.828822 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8935f31-2228-407e-85a5-3b140dbe2b82 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:00.829130 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8935f31-2228-407e-85a5-3b140dbe2b82 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:00.829368 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8935f31-2228-407e-85a5-3b140dbe2b82 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:00.831063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 75/140] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:00.839007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f5935b-7578-4ca3-bb3f-10854f38e806 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:00.839336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f5935b-7578-4ca3-bb3f-10854f38e806 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:00.839609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f5935b-7578-4ca3-bb3f-10854f38e806 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:00.861312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-85f5935b-7578-4ca3-bb3f-10854f38e806 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:00.862598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-85f5935b-7578-4ca3-bb3f-10854f38e806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:00.865848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:00.970016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 66/141] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:01.038457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8935f31-2228-407e-85a5-3b140dbe2b82 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:01.150336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8935f31-2228-407e-85a5-3b140dbe2b82 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:01.186649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7d0954-df95-4345-9b90-4cfb95c9c750 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:01.187695 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7d0954-df95-4345-9b90-4cfb95c9c750 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:01.188162 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7d0954-df95-4345-9b90-4cfb95c9c750 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:01.188621 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7d0954-df95-4345-9b90-4cfb95c9c750 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:01.189923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7d0954-df95-4345-9b90-4cfb95c9c750 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:01.191746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7d0954-df95-4345-9b90-4cfb95c9c750 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:01.193578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 76/142] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:01.199188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f5935b-7578-4ca3-bb3f-10854f38e806 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:01.326168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f5935b-7578-4ca3-bb3f-10854f38e806 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:01.371869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b187a3-b10a-4bb8-bda5-f1f199f7d335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:01.372837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b187a3-b10a-4bb8-bda5-f1f199f7d335 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:01.373320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b187a3-b10a-4bb8-bda5-f1f199f7d335 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:01.373775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b187a3-b10a-4bb8-bda5-f1f199f7d335 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:01.374655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b187a3-b10a-4bb8-bda5-f1f199f7d335 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:01.376353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b187a3-b10a-4bb8-bda5-f1f199f7d335 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:01.394497 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:01.465461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7d0954-df95-4345-9b90-4cfb95c9c750 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:01.469807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:01.490356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 67/143] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:26:01 2022] POST /identity/v3/users => generated 283 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:26:01.619493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7d0954-df95-4345-9b90-4cfb95c9c750 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:01.672473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5990b2-265e-4880-bccb-365e88bd5e26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:01.673136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5990b2-265e-4880-bccb-365e88bd5e26 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:01.673416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5990b2-265e-4880-bccb-365e88bd5e26 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:01.673684 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5990b2-265e-4880-bccb-365e88bd5e26 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:01.691156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5990b2-265e-4880-bccb-365e88bd5e26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-9b5990b2-265e-4880-bccb-365e88bd5e26 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:01.695866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:01.696705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:01.697471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 77/144] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:01 2022] GET /identity/v3/roles/reader => generated 84 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:01.707817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b187a3-b10a-4bb8-bda5-f1f199f7d335 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:01.834805 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b187a3-b10a-4bb8-bda5-f1f199f7d335 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:01.874282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf80e2e-920a-4a2b-9f58-eeb5457f3f1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:01.874776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf80e2e-920a-4a2b-9f58-eeb5457f3f1c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:01.875053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf80e2e-920a-4a2b-9f58-eeb5457f3f1c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:01.875321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf80e2e-920a-4a2b-9f58-eeb5457f3f1c None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:01.875989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf80e2e-920a-4a2b-9f58-eeb5457f3f1c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:01.876285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf80e2e-920a-4a2b-9f58-eeb5457f3f1c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:01.877651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf80e2e-920a-4a2b-9f58-eeb5457f3f1c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:01.888530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 68/145] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:01 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:01.894316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5990b2-265e-4880-bccb-365e88bd5e26 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:01.986933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:02.029239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5990b2-265e-4880-bccb-365e88bd5e26 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:02.071978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512c85f3-3511-41ee-bcb5-3f4c059b0280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:02.072618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512c85f3-3511-41ee-bcb5-3f4c059b0280 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:02.072897 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512c85f3-3511-41ee-bcb5-3f4c059b0280 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:02.073166 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512c85f3-3511-41ee-bcb5-3f4c059b0280 None admin] PATH_INFO: `/v3/users/6a30a0b87e6d4e788cbc5f35c9605f6b` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:02.078861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512c85f3-3511-41ee-bcb5-3f4c059b0280 None admin] RBAC: Authorizing `identity:get_user(user_id=6a30a0b87e6d4e788cbc5f35c9605f6b)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:02.080379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512c85f3-3511-41ee-bcb5-3f4c059b0280 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:02.082393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 78/146] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:01 2022] GET /identity/v3/users/6a30a0b87e6d4e788cbc5f35c9605f6b => generated 289 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:02.087762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf80e2e-920a-4a2b-9f58-eeb5457f3f1c None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:02.178293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:02.236874 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf80e2e-920a-4a2b-9f58-eeb5457f3f1c None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:02.284438 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b29677-ff6d-4a07-9f7a-575a5d6bf4b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:02.285253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b29677-ff6d-4a07-9f7a-575a5d6bf4b6 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:02.285601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b29677-ff6d-4a07-9f7a-575a5d6bf4b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:02.286083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b29677-ff6d-4a07-9f7a-575a5d6bf4b6 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:02.290907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b29677-ff6d-4a07-9f7a-575a5d6bf4b6 None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:02.292380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b29677-ff6d-4a07-9f7a-575a5d6bf4b6 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:02.294056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 69/147] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:02 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:02.300208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512c85f3-3511-41ee-bcb5-3f4c059b0280 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:02.442133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512c85f3-3511-41ee-bcb5-3f4c059b0280 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:02.477576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:02.482387 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3919c47-5167-4d6e-938b-b617e859677d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:02.483051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3919c47-5167-4d6e-938b-b617e859677d None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:02.483332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3919c47-5167-4d6e-938b-b617e859677d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:02.483600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3919c47-5167-4d6e-938b-b617e859677d None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db/users/6a30a0b87e6d4e788cbc5f35c9605f6b/roles/74e1fd5acabf47a7b788b2dc78d1fb3e` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:02.485594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3919c47-5167-4d6e-938b-b617e859677d None admin] RBAC: Authorizing `identity:create_grant(project_id=db44137c64a347cfb350dad24d36a3db, user_id=6a30a0b87e6d4e788cbc5f35c9605f6b, role_id=74e1fd5acabf47a7b788b2dc78d1fb3e)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:02.493225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3919c47-5167-4d6e-938b-b617e859677d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:02.562466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 79/148] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:26:02 2022] PUT /identity/v3/projects/db44137c64a347cfb350dad24d36a3db/users/6a30a0b87e6d4e788cbc5f35c9605f6b/roles/74e1fd5acabf47a7b788b2dc78d1fb3e => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:03.101748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af21f620-ea7c-43fa-ab68-e87cd0f19157 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:03.142809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af21f620-ea7c-43fa-ab68-e87cd0f19157 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:03.142809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af21f620-ea7c-43fa-ab68-e87cd0f19157 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:03.142809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 70/149] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:03.143066 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fdadf5-ae21-4ed8-b593-afdde5d9b5a3 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:03.143066 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fdadf5-ae21-4ed8-b593-afdde5d9b5a3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:03.143066 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fdadf5-ae21-4ed8-b593-afdde5d9b5a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:03.143066 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-a1fdadf5-ae21-4ed8-b593-afdde5d9b5a3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:03.143066 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a1fdadf5-ae21-4ed8-b593-afdde5d9b5a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:03.220641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 80/150] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:03.234370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54364974-27f4-4082-be8c-e046ac8a3bde None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:03.235061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54364974-27f4-4082-be8c-e046ac8a3bde None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:03.235061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54364974-27f4-4082-be8c-e046ac8a3bde None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:03.236459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 71/151] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:03.242299 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501c03e8-a67e-4a48-af8d-ebd1ab9d8567 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:03.242595 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501c03e8-a67e-4a48-af8d-ebd1ab9d8567 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:03.242862 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501c03e8-a67e-4a48-af8d-ebd1ab9d8567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:03.264580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-501c03e8-a67e-4a48-af8d-ebd1ab9d8567 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:03.265774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-501c03e8-a67e-4a48-af8d-ebd1ab9d8567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:03.346609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 81/152] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:03.542128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e372ec84-2668-472e-a962-76c8b7eece8d None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:03.542452 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e372ec84-2668-472e-a962-76c8b7eece8d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:03.542724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e372ec84-2668-472e-a962-76c8b7eece8d None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:03.544330 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 72/153] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:03.562624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9376768c-8aa7-40d5-9e8e-91af8ba0ae57 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:03.562904 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9376768c-8aa7-40d5-9e8e-91af8ba0ae57 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:03.563176 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9376768c-8aa7-40d5-9e8e-91af8ba0ae57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:03.591172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-9376768c-8aa7-40d5-9e8e-91af8ba0ae57 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:03.592374 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9376768c-8aa7-40d5-9e8e-91af8ba0ae57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:03.642674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:03.669879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 82/154] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:03.692397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e372ec84-2668-472e-a962-76c8b7eece8d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:03.800394 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e372ec84-2668-472e-a962-76c8b7eece8d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:03.825409 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:03.836367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0aeec3d-84e5-4e77-a058-186aedfb116d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:03.837025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aeec3d-84e5-4e77-a058-186aedfb116d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:03.837308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aeec3d-84e5-4e77-a058-186aedfb116d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:03.837584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aeec3d-84e5-4e77-a058-186aedfb116d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:03.849047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0aeec3d-84e5-4e77-a058-186aedfb116d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-c0aeec3d-84e5-4e77-a058-186aedfb116d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:03.853881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:03.854806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:03.855589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 73/155] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:03.861195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9376768c-8aa7-40d5-9e8e-91af8ba0ae57 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:03.976995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9376768c-8aa7-40d5-9e8e-91af8ba0ae57 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:04.013918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5171c410-ea20-4187-bab3-b6dde2b95d1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:04.014557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5171c410-ea20-4187-bab3-b6dde2b95d1e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:04.014834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5171c410-ea20-4187-bab3-b6dde2b95d1e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:04.015102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5171c410-ea20-4187-bab3-b6dde2b95d1e None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:04.015773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5171c410-ea20-4187-bab3-b6dde2b95d1e None admin] RBAC: Adding query filter params (name=member) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:04.016126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5171c410-ea20-4187-bab3-b6dde2b95d1e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:04.017540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5171c410-ea20-4187-bab3-b6dde2b95d1e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:04.023161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0aeec3d-84e5-4e77-a058-186aedfb116d None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:04.029488 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 83/156] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:04.040366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5171c410-ea20-4187-bab3-b6dde2b95d1e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:04.144331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:04.222098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5171c410-ea20-4187-bab3-b6dde2b95d1e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:04.247277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0aeec3d-84e5-4e77-a058-186aedfb116d None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:04.297328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8a94ca-8378-4c05-ae0b-93d229eb9daf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:04.297328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8a94ca-8378-4c05-ae0b-93d229eb9daf None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:04.297328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8a94ca-8378-4c05-ae0b-93d229eb9daf None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:04.297328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8a94ca-8378-4c05-ae0b-93d229eb9daf None admin] PATH_INFO: `/v3/users/18ccdd66160a423494d2172fb012e524` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:04.298519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8a94ca-8378-4c05-ae0b-93d229eb9daf None admin] RBAC: Authorizing `identity:get_user(user_id=18ccdd66160a423494d2172fb012e524)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:04.300379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8a94ca-8378-4c05-ae0b-93d229eb9daf None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:04.302320 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 84/157] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:04 2022] GET /identity/v3/users/18ccdd66160a423494d2172fb012e524 => generated 275 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:04.303037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d209b4fa-9c89-41f7-803c-e45bdc34f7b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:04.303779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8429a2-8b0b-4383-85d1-9fe10d05a66a None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:04.303984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d209b4fa-9c89-41f7-803c-e45bdc34f7b1 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:04.304268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8429a2-8b0b-4383-85d1-9fe10d05a66a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:04.304641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8429a2-8b0b-4383-85d1-9fe10d05a66a None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:04.304751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d209b4fa-9c89-41f7-803c-e45bdc34f7b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:04.305137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d209b4fa-9c89-41f7-803c-e45bdc34f7b1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:04.309897 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 85/158] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:04.309897 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8429a2-8b0b-4383-85d1-9fe10d05a66a None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:04.341495 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d209b4fa-9c89-41f7-803c-e45bdc34f7b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-d209b4fa-9c89-41f7-803c-e45bdc34f7b1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:04.349056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:04.349946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:04.350738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:04.350738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:04.350738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:04.350738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:04.350738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:04.350738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:04.350738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:04.350738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:04.350738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:04.350738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:04.350738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:04.350738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:04.355198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 74/159] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:26:04 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 333 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:04.361340 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d209b4fa-9c89-41f7-803c-e45bdc34f7b1 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:04.454378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:04.513294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8429a2-8b0b-4383-85d1-9fe10d05a66a None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:04.528920 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:04.555153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655581a0-c1eb-4d58-90ad-f17c754d4901 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:04.556095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655581a0-c1eb-4d58-90ad-f17c754d4901 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:04.557031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655581a0-c1eb-4d58-90ad-f17c754d4901 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:04.557462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655581a0-c1eb-4d58-90ad-f17c754d4901 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:04.571131 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655581a0-c1eb-4d58-90ad-f17c754d4901 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-655581a0-c1eb-4d58-90ad-f17c754d4901 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:04.577717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:04.578806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:04.579793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:04.579793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:04.579793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:04.579793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:04.579793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:04.579793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:04.579793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:04.579793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:04.579793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:04.579793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:04.579793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:04.579793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:04.582601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 86/160] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:26:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:04.583907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb45b86c-a161-4ffc-b913-a5a3802415dc None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:04.584392 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb45b86c-a161-4ffc-b913-a5a3802415dc None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:04.584834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb45b86c-a161-4ffc-b913-a5a3802415dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:04.592997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d209b4fa-9c89-41f7-803c-e45bdc34f7b1 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:04.609148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-bb45b86c-a161-4ffc-b913-a5a3802415dc None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:04.611089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-bb45b86c-a161-4ffc-b913-a5a3802415dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:04.688326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 87/161] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:04.689562 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb45b86c-a161-4ffc-b913-a5a3802415dc None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:04.706866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4b51c8-792c-4b45-9b13-e9037c331994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:04.707773 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4b51c8-792c-4b45-9b13-e9037c331994 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:04.708018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4b51c8-792c-4b45-9b13-e9037c331994 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:04.708296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4b51c8-792c-4b45-9b13-e9037c331994 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:04.801285 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb45b86c-a161-4ffc-b913-a5a3802415dc None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:04.818503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:04.841113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0101ea-3f5c-4754-b149-e6866bfcb950 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:04.845716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0101ea-3f5c-4754-b149-e6866bfcb950 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:04.845716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0101ea-3f5c-4754-b149-e6866bfcb950 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:04.845716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0101ea-3f5c-4754-b149-e6866bfcb950 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:04.845716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0101ea-3f5c-4754-b149-e6866bfcb950 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:04.845716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0101ea-3f5c-4754-b149-e6866bfcb950 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:04.845716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0101ea-3f5c-4754-b149-e6866bfcb950 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:04.856054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 88/162] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:04 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:04.857321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0101ea-3f5c-4754-b149-e6866bfcb950 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:04.881270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4b51c8-792c-4b45-9b13-e9037c331994 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:04.882587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4b51c8-792c-4b45-9b13-e9037c331994 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:04.958932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:05.029572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 75/163] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 669 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:05.030545 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4b51c8-792c-4b45-9b13-e9037c331994 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:05.108286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0101ea-3f5c-4754-b149-e6866bfcb950 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:05.148093 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4b51c8-792c-4b45-9b13-e9037c331994 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:05.199785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12ceb7b-05f4-411d-9044-7edcd9db68cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:05.200580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ceb7b-05f4-411d-9044-7edcd9db68cd None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:05.200874 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ceb7b-05f4-411d-9044-7edcd9db68cd None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:05.201144 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ceb7b-05f4-411d-9044-7edcd9db68cd None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:05.201810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12ceb7b-05f4-411d-9044-7edcd9db68cd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:05.202108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12ceb7b-05f4-411d-9044-7edcd9db68cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:05.203507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12ceb7b-05f4-411d-9044-7edcd9db68cd None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:05.206848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444ee73d-5f3f-4155-b45e-2866739cc3c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:05.208021 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444ee73d-5f3f-4155-b45e-2866739cc3c4 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:05.208295 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444ee73d-5f3f-4155-b45e-2866739cc3c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:05.208562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444ee73d-5f3f-4155-b45e-2866739cc3c4 None admin] PATH_INFO: `/v3/users/18ccdd66160a423494d2172fb012e524` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:05.209742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444ee73d-5f3f-4155-b45e-2866739cc3c4 None admin] RBAC: Authorizing `identity:get_user(user_id=18ccdd66160a423494d2172fb012e524)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:05.211198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444ee73d-5f3f-4155-b45e-2866739cc3c4 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:05.213125 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 76/164] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:05 2022] GET /identity/v3/users/18ccdd66160a423494d2172fb012e524 => generated 275 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:05.213903 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444ee73d-5f3f-4155-b45e-2866739cc3c4 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:05.221560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 89/165] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 365 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:05.230306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12ceb7b-05f4-411d-9044-7edcd9db68cd None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:05.239762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:05.287542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:05.344944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444ee73d-5f3f-4155-b45e-2866739cc3c4 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:05.385051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e9c507-b4f7-4a4d-98c7-701b633b2462 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:05.385608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e9c507-b4f7-4a4d-98c7-701b633b2462 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:05.385885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e9c507-b4f7-4a4d-98c7-701b633b2462 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:05.386180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e9c507-b4f7-4a4d-98c7-701b633b2462 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:05.388244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e9c507-b4f7-4a4d-98c7-701b633b2462 None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:05.389685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e9c507-b4f7-4a4d-98c7-701b633b2462 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:05.391348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 77/166] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:05 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:05.398222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e9c507-b4f7-4a4d-98c7-701b633b2462 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:05.496100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12ceb7b-05f4-411d-9044-7edcd9db68cd None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:05.496722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:05.510448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e9c507-b4f7-4a4d-98c7-701b633b2462 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:05.566291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce76a706-8097-47a9-831c-ba8e46c1b482 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:05.567480 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce76a706-8097-47a9-831c-ba8e46c1b482 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:05.567768 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce76a706-8097-47a9-831c-ba8e46c1b482 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:05.568010 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce76a706-8097-47a9-831c-ba8e46c1b482 None admin] PATH_INFO: `/v3/users/c73de3f5e3b94a3182c45d2476903a00` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:05.574195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce76a706-8097-47a9-831c-ba8e46c1b482 None admin] RBAC: Authorizing `identity:get_user(user_id=c73de3f5e3b94a3182c45d2476903a00)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:05.575846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce76a706-8097-47a9-831c-ba8e46c1b482 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:05.577483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 78/167] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:05 2022] GET /identity/v3/users/c73de3f5e3b94a3182c45d2476903a00 => generated 247 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:05.578493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce76a706-8097-47a9-831c-ba8e46c1b482 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:05.617747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d73712-3bf4-4e6f-a52f-f5f56bce2615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:05.618399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d73712-3bf4-4e6f-a52f-f5f56bce2615 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:05.618685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d73712-3bf4-4e6f-a52f-f5f56bce2615 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:05.618953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d73712-3bf4-4e6f-a52f-f5f56bce2615 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:05.690923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d73712-3bf4-4e6f-a52f-f5f56bce2615 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:05.692416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d73712-3bf4-4e6f-a52f-f5f56bce2615 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:05.718625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:05.741026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce76a706-8097-47a9-831c-ba8e46c1b482 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:05.777383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb48b90e-262d-46d5-8b35-055183e167e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:05.778284 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb48b90e-262d-46d5-8b35-055183e167e2 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:05.778722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb48b90e-262d-46d5-8b35-055183e167e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:05.779148 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb48b90e-262d-46d5-8b35-055183e167e2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:05.790642 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb48b90e-262d-46d5-8b35-055183e167e2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-eb48b90e-262d-46d5-8b35-055183e167e2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:05.795601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:05.796554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:05.797335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:05.797335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:05.797335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:05.797335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:05.797335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:05.797335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:05.797335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:05.797335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:05.797335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:05.797335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:05.797335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:05.797335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:05.800021 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 79/168] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:05.801203 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb48b90e-262d-46d5-8b35-055183e167e2 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:05.808204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 90/169] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 585 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:05.809250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d73712-3bf4-4e6f-a52f-f5f56bce2615 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:05.814016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:05.906192 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb48b90e-262d-46d5-8b35-055183e167e2 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:05.917484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d73712-3bf4-4e6f-a52f-f5f56bce2615 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:05.943877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e015bab6-1707-447c-b298-e9934e1a32e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:05.944557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e015bab6-1707-447c-b298-e9934e1a32e0 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:05.945267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e015bab6-1707-447c-b298-e9934e1a32e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:05.945267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e015bab6-1707-447c-b298-e9934e1a32e0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:05.946210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e015bab6-1707-447c-b298-e9934e1a32e0 None admin] RBAC: Adding query filter params (role.id=9912acfb698e41ebbce6a5147fafe896, scope.project.id=db44137c64a347cfb350dad24d36a3db, user.id=18ccdd66160a423494d2172fb012e524) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:05.946657 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e015bab6-1707-447c-b298-e9934e1a32e0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:05.947966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:05.949431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e015bab6-1707-447c-b298-e9934e1a32e0 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:05.955674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 80/170] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:05 2022] GET /identity/v3/role_assignments?user.id=18ccdd66160a423494d2172fb012e524&scope.project.id=db44137c64a347cfb350dad24d36a3db&role.id=9912acfb698e41ebbce6a5147fafe896 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:05.958086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b79b9f-d6f5-467e-88d4-2328d814a89c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:05.958951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b79b9f-d6f5-467e-88d4-2328d814a89c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:05.959392 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b79b9f-d6f5-467e-88d4-2328d814a89c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:05.959824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b79b9f-d6f5-467e-88d4-2328d814a89c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:05.963849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9c79f1-9030-4718-819a-f4a163e5eab4 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:05.964179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9c79f1-9030-4718-819a-f4a163e5eab4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:05.965293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9c79f1-9030-4718-819a-f4a163e5eab4 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:05.965920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 81/171] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:05 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:05.966789 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9c79f1-9030-4718-819a-f4a163e5eab4 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:06.045392 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b79b9f-d6f5-467e-88d4-2328d814a89c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:06.047330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b79b9f-d6f5-467e-88d4-2328d814a89c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:06.086354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:06.160457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9c79f1-9030-4718-819a-f4a163e5eab4 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:06.163981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 91/172] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:06.165098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b79b9f-d6f5-467e-88d4-2328d814a89c None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:06.195742 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:06.218062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7069755-b810-4c78-881f-548944fb6a09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:06.219030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7069755-b810-4c78-881f-548944fb6a09 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.219030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7069755-b810-4c78-881f-548944fb6a09 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.219324 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7069755-b810-4c78-881f-548944fb6a09 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.219933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7069755-b810-4c78-881f-548944fb6a09 None admin] RBAC: Adding query filter params (name=member) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:06.220234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7069755-b810-4c78-881f-548944fb6a09 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:06.221647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7069755-b810-4c78-881f-548944fb6a09 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:06.238780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 82/173] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:06.244715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65030f81-dc93-4912-9155-6ae557690c8c None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.244715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65030f81-dc93-4912-9155-6ae557690c8c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.244715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65030f81-dc93-4912-9155-6ae557690c8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.275353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-65030f81-dc93-4912-9155-6ae557690c8c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:06.276597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-65030f81-dc93-4912-9155-6ae557690c8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:06.291550 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b79b9f-d6f5-467e-88d4-2328d814a89c None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:06.333107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afca47b7-f651-41e1-bcdc-3a8a9de1383d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:06.334202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afca47b7-f651-41e1-bcdc-3a8a9de1383d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.334765 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afca47b7-f651-41e1-bcdc-3a8a9de1383d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.335352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afca47b7-f651-41e1-bcdc-3a8a9de1383d None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.336843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afca47b7-f651-41e1-bcdc-3a8a9de1383d None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:06.338618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afca47b7-f651-41e1-bcdc-3a8a9de1383d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:06.340585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 92/174] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:06 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:06.341649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afca47b7-f651-41e1-bcdc-3a8a9de1383d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:06.373135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 83/175] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:06.373991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65030f81-dc93-4912-9155-6ae557690c8c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:06.441114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:06.489262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afca47b7-f651-41e1-bcdc-3a8a9de1383d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:06.492831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:06.514114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65030f81-dc93-4912-9155-6ae557690c8c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:06.540580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615dd923-fe9d-460d-b81d-be109133bbd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:06.541208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615dd923-fe9d-460d-b81d-be109133bbd5 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.541481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615dd923-fe9d-460d-b81d-be109133bbd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.541748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615dd923-fe9d-460d-b81d-be109133bbd5 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.542900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615dd923-fe9d-460d-b81d-be109133bbd5 None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:06.544345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615dd923-fe9d-460d-b81d-be109133bbd5 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:06.545956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 93/176] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:06 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:06.549926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976a559-6bbc-4082-b841-dec6c3b96dc1 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.549926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976a559-6bbc-4082-b841-dec6c3b96dc1 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.549926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976a559-6bbc-4082-b841-dec6c3b96dc1 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.549926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 94/177] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:06 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:06.549926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976a559-6bbc-4082-b841-dec6c3b96dc1 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:06.563056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6e0079-24c4-43d8-95ce-2e4729d0d556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:06.563777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6e0079-24c4-43d8-95ce-2e4729d0d556 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.564124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6e0079-24c4-43d8-95ce-2e4729d0d556 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.564489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6e0079-24c4-43d8-95ce-2e4729d0d556 None admin] PATH_INFO: `/v3/users/18ccdd66160a423494d2172fb012e524` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.565660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6e0079-24c4-43d8-95ce-2e4729d0d556 None admin] RBAC: Authorizing `identity:get_user(user_id=18ccdd66160a423494d2172fb012e524)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:06.567766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6e0079-24c4-43d8-95ce-2e4729d0d556 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:06.569570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 84/178] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:06 2022] GET /identity/v3/users/18ccdd66160a423494d2172fb012e524 => generated 275 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:06.572715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943615f0-69a1-428b-a9cd-1114876dc5e0 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.572998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943615f0-69a1-428b-a9cd-1114876dc5e0 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.573270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943615f0-69a1-428b-a9cd-1114876dc5e0 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.574859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 85/179] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:06 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:06.576013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6445675c-38a4-42ec-a3d7-ffce316d8834 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.576290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6445675c-38a4-42ec-a3d7-ffce316d8834 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.576559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6445675c-38a4-42ec-a3d7-ffce316d8834 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.577858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 86/180] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:06 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:06.580891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6445675c-38a4-42ec-a3d7-ffce316d8834 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:06.671893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976a559-6bbc-4082-b841-dec6c3b96dc1 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:06.696593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6445675c-38a4-42ec-a3d7-ffce316d8834 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:06.708408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcbbcfc-dc8a-44da-a98a-77efc992326e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:06.709096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcbbcfc-dc8a-44da-a98a-77efc992326e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.709379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcbbcfc-dc8a-44da-a98a-77efc992326e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.709649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcbbcfc-dc8a-44da-a98a-77efc992326e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.710401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcbbcfc-dc8a-44da-a98a-77efc992326e None admin] RBAC: Adding query filter params (role.id=d02a8e6c49b74aaa9403f8bd535b90bf, scope.project.id=db44137c64a347cfb350dad24d36a3db, user.id=18ccdd66160a423494d2172fb012e524) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:06.710702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcbbcfc-dc8a-44da-a98a-77efc992326e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:06.712189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcbbcfc-dc8a-44da-a98a-77efc992326e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:06.718474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 95/181] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:06 2022] GET /identity/v3/role_assignments?user.id=18ccdd66160a423494d2172fb012e524&scope.project.id=db44137c64a347cfb350dad24d36a3db&role.id=d02a8e6c49b74aaa9403f8bd535b90bf => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:06.719618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98962d2-7c44-43e7-a4b2-22f1d9c787da None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.719898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98962d2-7c44-43e7-a4b2-22f1d9c787da None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.720165 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98962d2-7c44-43e7-a4b2-22f1d9c787da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.739426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:06.740554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-a98962d2-7c44-43e7-a4b2-22f1d9c787da None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:06.741704 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a98962d2-7c44-43e7-a4b2-22f1d9c787da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:06.745091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510b40b1-9d2a-45df-b208-afde625b8d45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:06.745723 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510b40b1-9d2a-45df-b208-afde625b8d45 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.746007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510b40b1-9d2a-45df-b208-afde625b8d45 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.746320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510b40b1-9d2a-45df-b208-afde625b8d45 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.747069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510b40b1-9d2a-45df-b208-afde625b8d45 None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=db44137c64a347cfb350dad24d36a3db, user.id=c73de3f5e3b94a3182c45d2476903a00) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:06.747381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510b40b1-9d2a-45df-b208-afde625b8d45 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:06.748887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510b40b1-9d2a-45df-b208-afde625b8d45 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:06.759814 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 87/182] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:06 2022] GET /identity/v3/role_assignments?user.id=c73de3f5e3b94a3182c45d2476903a00&scope.project.id=db44137c64a347cfb350dad24d36a3db&role.id=43ca25467bc6484da3a50508efb79230 => generated 608 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:06.759814 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510b40b1-9d2a-45df-b208-afde625b8d45 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:06.767786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:06.830481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 96/183] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:06.832311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fe2f6f-23f2-4075-8c89-124147c54489 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.838764 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fe2f6f-23f2-4075-8c89-124147c54489 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.839192 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fe2f6f-23f2-4075-8c89-124147c54489 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.881045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-57fe2f6f-23f2-4075-8c89-124147c54489 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:06.886182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-57fe2f6f-23f2-4075-8c89-124147c54489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:06.911226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510b40b1-9d2a-45df-b208-afde625b8d45 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:06.960177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8c3132-7fd3-4faa-a682-12087039695a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:06.960921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8c3132-7fd3-4faa-a682-12087039695a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.961197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8c3132-7fd3-4faa-a682-12087039695a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.961468 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8c3132-7fd3-4faa-a682-12087039695a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:06.981948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 97/184] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:06.984282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39fa7fa-1ed4-49c7-97c8-782592e9c3a8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:06.984564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39fa7fa-1ed4-49c7-97c8-782592e9c3a8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:06.984836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39fa7fa-1ed4-49c7-97c8-782592e9c3a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:07.004383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-b39fa7fa-1ed4-49c7-97c8-782592e9c3a8 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:07.005539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b39fa7fa-1ed4-49c7-97c8-782592e9c3a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:07.036958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:07.043904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8c3132-7fd3-4faa-a682-12087039695a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:07.045760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8c3132-7fd3-4faa-a682-12087039695a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:07.091909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 98/185] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:07.092843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39fa7fa-1ed4-49c7-97c8-782592e9c3a8 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:07.116256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:07.178552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 88/186] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:07.185235 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8c3132-7fd3-4faa-a682-12087039695a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:07.240314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39fa7fa-1ed4-49c7-97c8-782592e9c3a8 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:07.286300 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d74385b-b93c-407a-84e1-bf95ff262d95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:07.287021 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d74385b-b93c-407a-84e1-bf95ff262d95 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:07.287413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d74385b-b93c-407a-84e1-bf95ff262d95 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:07.287689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d74385b-b93c-407a-84e1-bf95ff262d95 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:07.289911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:07.295895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8c3132-7fd3-4faa-a682-12087039695a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:07.301718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d74385b-b93c-407a-84e1-bf95ff262d95 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-5d74385b-b93c-407a-84e1-bf95ff262d95 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:07.307745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:07.308630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:07.309423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 99/187] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:26:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:07.310064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d74385b-b93c-407a-84e1-bf95ff262d95 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:07.341793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270ebcf9-5a8d-4e77-ac5c-d819c62067c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:07.342691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270ebcf9-5a8d-4e77-ac5c-d819c62067c3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:07.343137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270ebcf9-5a8d-4e77-ac5c-d819c62067c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:07.343563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270ebcf9-5a8d-4e77-ac5c-d819c62067c3 None admin] PATH_INFO: `/v3/projects/86c05145fc0e4ca79b274eac020a148c` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:07.344832 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270ebcf9-5a8d-4e77-ac5c-d819c62067c3 None admin] RBAC: Authorizing `identity:get_project(project_id=86c05145fc0e4ca79b274eac020a148c)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:07.346451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270ebcf9-5a8d-4e77-ac5c-d819c62067c3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:07.348262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 89/188] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:07 2022] GET /identity/v3/projects/86c05145fc0e4ca79b274eac020a148c => generated 312 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:07.349154 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270ebcf9-5a8d-4e77-ac5c-d819c62067c3 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:07.401587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:07.430929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d74385b-b93c-407a-84e1-bf95ff262d95 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:07.455848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270ebcf9-5a8d-4e77-ac5c-d819c62067c3 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:07.465179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de96335a-0ec2-4a1a-a324-523fa38e1b39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:07.465813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de96335a-0ec2-4a1a-a324-523fa38e1b39 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:07.466093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de96335a-0ec2-4a1a-a324-523fa38e1b39 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:07.466395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de96335a-0ec2-4a1a-a324-523fa38e1b39 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:07.476952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de96335a-0ec2-4a1a-a324-523fa38e1b39 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-de96335a-0ec2-4a1a-a324-523fa38e1b39 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:07.482122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:07.483077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:07.483865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 100/189] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:26:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:07.484502 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de96335a-0ec2-4a1a-a324-523fa38e1b39 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:07.492329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e95909-77ad-4fd5-aa4c-2f406132764a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:07.492932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e95909-77ad-4fd5-aa4c-2f406132764a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:07.493202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e95909-77ad-4fd5-aa4c-2f406132764a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:07.493469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e95909-77ad-4fd5-aa4c-2f406132764a None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:07.494121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e95909-77ad-4fd5-aa4c-2f406132764a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:07.494446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e95909-77ad-4fd5-aa4c-2f406132764a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:07.495803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e95909-77ad-4fd5-aa4c-2f406132764a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:07.509137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 90/190] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:07.509137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e95909-77ad-4fd5-aa4c-2f406132764a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:07.535186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:07.583597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de96335a-0ec2-4a1a-a324-523fa38e1b39 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:07.607673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e95909-77ad-4fd5-aa4c-2f406132764a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:07.616584 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce7a10-63da-45c8-a0c8-60931b5edc57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:07.617217 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce7a10-63da-45c8-a0c8-60931b5edc57 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:07.617497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce7a10-63da-45c8-a0c8-60931b5edc57 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:07.617767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce7a10-63da-45c8-a0c8-60931b5edc57 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:07.620321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:07.628471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ce7a10-63da-45c8-a0c8-60931b5edc57 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-91ce7a10-63da-45c8-a0c8-60931b5edc57 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:07.634122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:07.634993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:07.635799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 101/191] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:07 2022] GET /identity/v3/roles/reader => generated 84 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:07.636442 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce7a10-63da-45c8-a0c8-60931b5edc57 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:07.648286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8184b34d-ac78-4f77-b0f9-b4475a312ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:07.648905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8184b34d-ac78-4f77-b0f9-b4475a312ecf None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:07.649168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8184b34d-ac78-4f77-b0f9-b4475a312ecf None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:07.649434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8184b34d-ac78-4f77-b0f9-b4475a312ecf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:07.650183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8184b34d-ac78-4f77-b0f9-b4475a312ecf None admin] RBAC: Adding query filter params (role.id=9912acfb698e41ebbce6a5147fafe896, scope.project.id=86c05145fc0e4ca79b274eac020a148c, user.id=18ccdd66160a423494d2172fb012e524) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:07.650490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8184b34d-ac78-4f77-b0f9-b4475a312ecf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:07.651921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8184b34d-ac78-4f77-b0f9-b4475a312ecf None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:07.657531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 91/192] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:07 2022] GET /identity/v3/role_assignments?user.id=18ccdd66160a423494d2172fb012e524&scope.project.id=86c05145fc0e4ca79b274eac020a148c&role.id=9912acfb698e41ebbce6a5147fafe896 => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:07.659108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8184b34d-ac78-4f77-b0f9-b4475a312ecf None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:07.750730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce7a10-63da-45c8-a0c8-60931b5edc57 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:07.756064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:07.766571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8184b34d-ac78-4f77-b0f9-b4475a312ecf None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:07.798479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8660dc09-06a0-4a76-9610-ce19f20db798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:07.799105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8660dc09-06a0-4a76-9610-ce19f20db798 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:07.799384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8660dc09-06a0-4a76-9610-ce19f20db798 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:07.799650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8660dc09-06a0-4a76-9610-ce19f20db798 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:07.801079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8660dc09-06a0-4a76-9610-ce19f20db798 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:07.802489 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8660dc09-06a0-4a76-9610-ce19f20db798 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:07.804413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 102/193] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:07.804805 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8660dc09-06a0-4a76-9610-ce19f20db798 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:07.808673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5e5088-ad16-414d-aa24-018d3c4c7ffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:07.809477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5e5088-ad16-414d-aa24-018d3c4c7ffd None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:07.810572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5e5088-ad16-414d-aa24-018d3c4c7ffd None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:07.810850 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5e5088-ad16-414d-aa24-018d3c4c7ffd None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:07.812285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5e5088-ad16-414d-aa24-018d3c4c7ffd None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:07.812285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5e5088-ad16-414d-aa24-018d3c4c7ffd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:07.813214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5e5088-ad16-414d-aa24-018d3c4c7ffd None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:07.822139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 92/194] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:07.826851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5e5088-ad16-414d-aa24-018d3c4c7ffd None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:07.888812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:07.926546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8660dc09-06a0-4a76-9610-ce19f20db798 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:07.934336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5e5088-ad16-414d-aa24-018d3c4c7ffd None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:07.959254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40ce65b-963d-4990-bccc-592725d3ed15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:07.959880 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40ce65b-963d-4990-bccc-592725d3ed15 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:07.960159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40ce65b-963d-4990-bccc-592725d3ed15 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:07.960429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40ce65b-963d-4990-bccc-592725d3ed15 None admin] PATH_INFO: `/v3/users/1a8653bb588847d6a601967d2bae7a36` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:07.967152 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae00d1a7-d469-484e-a0c8-e980220f4e67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:07.967750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae00d1a7-d469-484e-a0c8-e980220f4e67 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:07.968023 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae00d1a7-d469-484e-a0c8-e980220f4e67 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:07.968294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae00d1a7-d469-484e-a0c8-e980220f4e67 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:07.968946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae00d1a7-d469-484e-a0c8-e980220f4e67 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:07.969244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae00d1a7-d469-484e-a0c8-e980220f4e67 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:07.970639 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae00d1a7-d469-484e-a0c8-e980220f4e67 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:07.973948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40ce65b-963d-4990-bccc-592725d3ed15 None admin] RBAC: Authorizing `identity:get_user(user_id=1a8653bb588847d6a601967d2bae7a36)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:07.975433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40ce65b-963d-4990-bccc-592725d3ed15 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:07.977036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 103/195] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:07 2022] GET /identity/v3/users/1a8653bb588847d6a601967d2bae7a36 => generated 283 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:07.977779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40ce65b-963d-4990-bccc-592725d3ed15 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:07.979445 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 93/196] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:07 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:07.980101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae00d1a7-d469-484e-a0c8-e980220f4e67 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:07.996530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:08.075959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40ce65b-963d-4990-bccc-592725d3ed15 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:08.080118 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae00d1a7-d469-484e-a0c8-e980220f4e67 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:08.099178 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:08.110029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da95a3fd-000c-4414-8e8d-23c39b1e3960 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:08.110656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da95a3fd-000c-4414-8e8d-23c39b1e3960 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:08.110929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da95a3fd-000c-4414-8e8d-23c39b1e3960 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:08.114696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da95a3fd-000c-4414-8e8d-23c39b1e3960 None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:08.114696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da95a3fd-000c-4414-8e8d-23c39b1e3960 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:08.114696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da95a3fd-000c-4414-8e8d-23c39b1e3960 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:08.115089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9de4a0-a0db-40ee-a369-f4437e46fd2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:08.115089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9de4a0-a0db-40ee-a369-f4437e46fd2e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:08.115257 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9de4a0-a0db-40ee-a369-f4437e46fd2e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:08.115430 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9de4a0-a0db-40ee-a369-f4437e46fd2e None admin] PATH_INFO: `/v3/users/1a8653bb588847d6a601967d2bae7a36` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:08.116583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9de4a0-a0db-40ee-a369-f4437e46fd2e None admin] RBAC: Authorizing `identity:get_user(user_id=1a8653bb588847d6a601967d2bae7a36)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:08.117992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9de4a0-a0db-40ee-a369-f4437e46fd2e None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:08.119788 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 94/197] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:07 2022] GET /identity/v3/users/1a8653bb588847d6a601967d2bae7a36 => generated 283 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:08.120320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9de4a0-a0db-40ee-a369-f4437e46fd2e None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:08.202402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:08.219885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9de4a0-a0db-40ee-a369-f4437e46fd2e None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:08.236028 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 104/198] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:26:07 2022] POST /identity/v3/users => generated 297 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:26:08.236951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da95a3fd-000c-4414-8e8d-23c39b1e3960 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:08.269167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c163bea-cddc-4802-ac74-ed609f2a51d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:08.270898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c163bea-cddc-4802-ac74-ed609f2a51d8 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:08.271490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c163bea-cddc-4802-ac74-ed609f2a51d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:08.271790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c163bea-cddc-4802-ac74-ed609f2a51d8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:08.350497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da95a3fd-000c-4414-8e8d-23c39b1e3960 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:08.353259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c163bea-cddc-4802-ac74-ed609f2a51d8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:08.354845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c163bea-cddc-4802-ac74-ed609f2a51d8 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:08.393960 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bad105-4745-41a9-8aa9-9118eefdd3cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:08.396206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bad105-4745-41a9-8aa9-9118eefdd3cf None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:08.396206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bad105-4745-41a9-8aa9-9118eefdd3cf None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:08.396206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bad105-4745-41a9-8aa9-9118eefdd3cf None admin] PATH_INFO: `/v3/users/6a30a0b87e6d4e788cbc5f35c9605f6b` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:08.396947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bad105-4745-41a9-8aa9-9118eefdd3cf None admin] RBAC: Authorizing `identity:get_user(user_id=6a30a0b87e6d4e788cbc5f35c9605f6b)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:08.398904 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bad105-4745-41a9-8aa9-9118eefdd3cf None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:08.400883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 105/199] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:08 2022] GET /identity/v3/users/6a30a0b87e6d4e788cbc5f35c9605f6b => generated 289 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:08.401744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bad105-4745-41a9-8aa9-9118eefdd3cf None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:08.463085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:08.465751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:08.497230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 95/200] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:08.498126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c163bea-cddc-4802-ac74-ed609f2a51d8 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:08.534793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bad105-4745-41a9-8aa9-9118eefdd3cf None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:08.585569 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2457d53f-22c2-4183-b5bd-9a244d090533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:08.586325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2457d53f-22c2-4183-b5bd-9a244d090533 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:08.586702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2457d53f-22c2-4183-b5bd-9a244d090533 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:08.586973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2457d53f-22c2-4183-b5bd-9a244d090533 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:08.632587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c163bea-cddc-4802-ac74-ed609f2a51d8 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:08.678444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d98486-8df6-4970-a8ec-9806ab9a7f8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:08.678944 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2457d53f-22c2-4183-b5bd-9a244d090533 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:08.679061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d98486-8df6-4970-a8ec-9806ab9a7f8b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:08.679328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d98486-8df6-4970-a8ec-9806ab9a7f8b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:08.679603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d98486-8df6-4970-a8ec-9806ab9a7f8b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:08.680427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2457d53f-22c2-4183-b5bd-9a244d090533 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:08.728154 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:08.736657 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:08.757624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d98486-8df6-4970-a8ec-9806ab9a7f8b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:08.760318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d98486-8df6-4970-a8ec-9806ab9a7f8b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:08.785709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 106/201] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:08.791048 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2457d53f-22c2-4183-b5bd-9a244d090533 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:08.873195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 96/202] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:08.873949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d98486-8df6-4970-a8ec-9806ab9a7f8b None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:08.899607 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2457d53f-22c2-4183-b5bd-9a244d090533 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:08.947642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305b6509-4c24-4de6-9f05-a89e30ad1141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:08.948278 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305b6509-4c24-4de6-9f05-a89e30ad1141 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:08.948561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305b6509-4c24-4de6-9f05-a89e30ad1141 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:08.949095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305b6509-4c24-4de6-9f05-a89e30ad1141 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:08.950852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305b6509-4c24-4de6-9f05-a89e30ad1141 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:08.952575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305b6509-4c24-4de6-9f05-a89e30ad1141 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:08.956615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 107/203] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:08 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:08.957941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305b6509-4c24-4de6-9f05-a89e30ad1141 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:08.972079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:08.986585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:08.996849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d98486-8df6-4970-a8ec-9806ab9a7f8b None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:09.045680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ab7719-89a4-4496-b448-b2346556921b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:09.046637 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ab7719-89a4-4496-b448-b2346556921b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:09.047371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ab7719-89a4-4496-b448-b2346556921b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:09.047790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ab7719-89a4-4496-b448-b2346556921b None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:09.049933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ab7719-89a4-4496-b448-b2346556921b None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:09.053424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ab7719-89a4-4496-b448-b2346556921b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:09.055770 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 97/204] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:08 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:09.056824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ab7719-89a4-4496-b448-b2346556921b None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:09.092432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305b6509-4c24-4de6-9f05-a89e30ad1141 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:09.130238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc071fcb-06e1-483d-91b2-20eb3b270305 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:09.131599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc071fcb-06e1-483d-91b2-20eb3b270305 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:09.131599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc071fcb-06e1-483d-91b2-20eb3b270305 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:09.131996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc071fcb-06e1-483d-91b2-20eb3b270305 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:09.133228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc071fcb-06e1-483d-91b2-20eb3b270305 None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:09.134820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc071fcb-06e1-483d-91b2-20eb3b270305 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:09.136820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 108/205] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:08 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:09.137739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc071fcb-06e1-483d-91b2-20eb3b270305 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:09.199563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ab7719-89a4-4496-b448-b2346556921b None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:09.232088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:09.249904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7a5145-4664-4256-b2d4-adf55155ac02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:09.250720 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7a5145-4664-4256-b2d4-adf55155ac02 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:09.250987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc071fcb-06e1-483d-91b2-20eb3b270305 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:09.251316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7a5145-4664-4256-b2d4-adf55155ac02 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:09.251727 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7a5145-4664-4256-b2d4-adf55155ac02 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:09.252762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7a5145-4664-4256-b2d4-adf55155ac02 None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=5c50437619304d0d857f65feed1d19e3, user.id=1a8653bb588847d6a601967d2bae7a36) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:09.253268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7a5145-4664-4256-b2d4-adf55155ac02 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:09.256422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7a5145-4664-4256-b2d4-adf55155ac02 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:09.262527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 98/206] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:09 2022] GET /identity/v3/role_assignments?user.id=1a8653bb588847d6a601967d2bae7a36&scope.project.id=5c50437619304d0d857f65feed1d19e3&role.id=43ca25467bc6484da3a50508efb79230 => generated 259 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:09.264685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7a5145-4664-4256-b2d4-adf55155ac02 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:09.291514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:09.330672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8151bc-5282-495a-a2f9-28b9541af92e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:09.331295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8151bc-5282-495a-a2f9-28b9541af92e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:09.331574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8151bc-5282-495a-a2f9-28b9541af92e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:09.331844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8151bc-5282-495a-a2f9-28b9541af92e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:09.332667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8151bc-5282-495a-a2f9-28b9541af92e None admin] RBAC: Adding query filter params (role.id=d02a8e6c49b74aaa9403f8bd535b90bf, scope.project.id=5c50437619304d0d857f65feed1d19e3, user.id=1a8653bb588847d6a601967d2bae7a36) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:09.332979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8151bc-5282-495a-a2f9-28b9541af92e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:09.337035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8151bc-5282-495a-a2f9-28b9541af92e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:09.353286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 109/207] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:09 2022] GET /identity/v3/role_assignments?user.id=1a8653bb588847d6a601967d2bae7a36&scope.project.id=5c50437619304d0d857f65feed1d19e3&role.id=d02a8e6c49b74aaa9403f8bd535b90bf => generated 259 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:09.467719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7a5145-4664-4256-b2d4-adf55155ac02 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:09.508565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2805b5b-6963-40e6-936c-5d4bc845d479 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:09.509106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2805b5b-6963-40e6-936c-5d4bc845d479 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:09.509384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2805b5b-6963-40e6-936c-5d4bc845d479 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:09.509663 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2805b5b-6963-40e6-936c-5d4bc845d479 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:09.510438 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2805b5b-6963-40e6-936c-5d4bc845d479 None admin] RBAC: Adding query filter params (role.id=74e1fd5acabf47a7b788b2dc78d1fb3e, scope.project.id=db44137c64a347cfb350dad24d36a3db, user.id=6a30a0b87e6d4e788cbc5f35c9605f6b) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:09.510748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2805b5b-6963-40e6-936c-5d4bc845d479 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:09.512219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2805b5b-6963-40e6-936c-5d4bc845d479 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:09.518907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 99/208] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:09 2022] GET /identity/v3/role_assignments?user.id=6a30a0b87e6d4e788cbc5f35c9605f6b&scope.project.id=db44137c64a347cfb350dad24d36a3db&role.id=74e1fd5acabf47a7b788b2dc78d1fb3e => generated 608 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:10.462065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d19b3c9-7852-46ee-ac37-12fa0eda01d9 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:10.462674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d19b3c9-7852-46ee-ac37-12fa0eda01d9 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:10.462674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d19b3c9-7852-46ee-ac37-12fa0eda01d9 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:10.463980 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 110/209] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:10.469295 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cd21e1-b847-45e6-a7a3-d98e8413bcac None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:10.469690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cd21e1-b847-45e6-a7a3-d98e8413bcac None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:10.469838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cd21e1-b847-45e6-a7a3-d98e8413bcac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:10.482953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e231a74f-1bfb-4ef9-a4d1-4d5f29ce06c9 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:10.483240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e231a74f-1bfb-4ef9-a4d1-4d5f29ce06c9 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:10.483499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e231a74f-1bfb-4ef9-a4d1-4d5f29ce06c9 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:10.485082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 111/210] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:10.491184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469d8255-fe50-42b7-9ae6-962cf4818674 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:10.491463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469d8255-fe50-42b7-9ae6-962cf4818674 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:10.491732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469d8255-fe50-42b7-9ae6-962cf4818674 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:10.493570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-52cd21e1-b847-45e6-a7a3-d98e8413bcac None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:10.495031 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-52cd21e1-b847-45e6-a7a3-d98e8413bcac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:10.511062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-469d8255-fe50-42b7-9ae6-962cf4818674 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:10.512314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-469d8255-fe50-42b7-9ae6-962cf4818674 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:10.571507 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 100/211] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:10.593444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 112/212] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:26:10.837341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cd21e1-b847-45e6-a7a3-d98e8413bcac None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:10.848185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:10.867489 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469d8255-fe50-42b7-9ae6-962cf4818674 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:10.959522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:10.964545 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cd21e1-b847-45e6-a7a3-d98e8413bcac None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:10.988712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469d8255-fe50-42b7-9ae6-962cf4818674 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:11.002676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacc287b-51f2-415e-9a6d-c3e74a9798c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:11.003635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacc287b-51f2-415e-9a6d-c3e74a9798c0 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:11.004804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacc287b-51f2-415e-9a6d-c3e74a9798c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:11.005270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacc287b-51f2-415e-9a6d-c3e74a9798c0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:11.016465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacc287b-51f2-415e-9a6d-c3e74a9798c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-aacc287b-51f2-415e-9a6d-c3e74a9798c0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:11.021532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:11.022444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:11.023234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:11.023234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:11.023234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:11.023234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:11.023234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:11.023234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:11.023234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:11.023234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:11.023234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:11.023234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:11.023234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:11.023234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:11.025787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 101/213] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:11.029570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf03c76-434f-40b0-9d48-7b3629762de7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:11.030482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf03c76-434f-40b0-9d48-7b3629762de7 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:11.031537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacc287b-51f2-415e-9a6d-c3e74a9798c0 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:11.032429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf03c76-434f-40b0-9d48-7b3629762de7 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:11.033244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf03c76-434f-40b0-9d48-7b3629762de7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:11.034655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf03c76-434f-40b0-9d48-7b3629762de7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:11.037035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf03c76-434f-40b0-9d48-7b3629762de7 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:11.039000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 113/214] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:11.045366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf03c76-434f-40b0-9d48-7b3629762de7 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:11.107489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:11.152903 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacc287b-51f2-415e-9a6d-c3e74a9798c0 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:11.163380 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf03c76-434f-40b0-9d48-7b3629762de7 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:11.201083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8409ad-bab5-490e-86f2-825f836fee10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:11.202340 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8409ad-bab5-490e-86f2-825f836fee10 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:11.203190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8409ad-bab5-490e-86f2-825f836fee10 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:11.203919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8409ad-bab5-490e-86f2-825f836fee10 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:11.204986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8409ad-bab5-490e-86f2-825f836fee10 None admin] RBAC: Adding query filter params (name=member) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:11.205501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8409ad-bab5-490e-86f2-825f836fee10 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:11.208108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8409ad-bab5-490e-86f2-825f836fee10 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:11.208414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2622426-e46d-48c2-aeed-9c817b8511de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:11.209149 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2622426-e46d-48c2-aeed-9c817b8511de None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:11.209535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2622426-e46d-48c2-aeed-9c817b8511de None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:11.209900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2622426-e46d-48c2-aeed-9c817b8511de None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:11.210684 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2622426-e46d-48c2-aeed-9c817b8511de None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:11.212224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2622426-e46d-48c2-aeed-9c817b8511de None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:11.218286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 102/215] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:11.224377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8409ad-bab5-490e-86f2-825f836fee10 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:11.317748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:11.330179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 114/216] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:26:11 2022] POST /identity/v3/users => generated 297 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:26:11.335974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8409ad-bab5-490e-86f2-825f836fee10 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:11.366783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5659180-fe94-4c2f-abb9-852a47d8135a None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:11.367089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5659180-fe94-4c2f-abb9-852a47d8135a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:11.367268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5659180-fe94-4c2f-abb9-852a47d8135a None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:11.368788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 115/217] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:26:11.372880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:11.374673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf14f316-05c5-450a-8128-1ec4546a0a50 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:11.374959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf14f316-05c5-450a-8128-1ec4546a0a50 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:11.375232 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf14f316-05c5-450a-8128-1ec4546a0a50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:11.378622 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc6b269-e12b-471e-b5e6-46dd61706e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:11.379570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc6b269-e12b-471e-b5e6-46dd61706e17 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:11.379850 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc6b269-e12b-471e-b5e6-46dd61706e17 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:11.380125 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc6b269-e12b-471e-b5e6-46dd61706e17 None admin] PATH_INFO: `/v3/users/3c47c2755dfc48c38c37c96e25db3829` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:11.397022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-bf14f316-05c5-450a-8128-1ec4546a0a50 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:11.398877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-bf14f316-05c5-450a-8128-1ec4546a0a50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:11.402246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc6b269-e12b-471e-b5e6-46dd61706e17 None admin] RBAC: Authorizing `identity:get_user(user_id=3c47c2755dfc48c38c37c96e25db3829)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:11.403738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc6b269-e12b-471e-b5e6-46dd61706e17 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:11.409439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 103/218] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:11 2022] GET /identity/v3/users/3c47c2755dfc48c38c37c96e25db3829 => generated 297 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:11.413729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc6b269-e12b-471e-b5e6-46dd61706e17 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:11.487412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 116/219] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:26:11.489063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f857880-8bc1-4d21-9f67-3734f3ae1895 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:11.489417 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f857880-8bc1-4d21-9f67-3734f3ae1895 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:11.489692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f857880-8bc1-4d21-9f67-3734f3ae1895 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:11.492821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 117/220] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:11.499363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb1d63b-a6a6-4d48-890d-b293d29af5da None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:11.499491 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb1d63b-a6a6-4d48-890d-b293d29af5da None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:11.499656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb1d63b-a6a6-4d48-890d-b293d29af5da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:11.522528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-8eb1d63b-a6a6-4d48-890d-b293d29af5da None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:11.523747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8eb1d63b-a6a6-4d48-890d-b293d29af5da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:11.538944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc6b269-e12b-471e-b5e6-46dd61706e17 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:11.586244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0579a65d-dfff-4959-9634-12c9199c2e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:11.587023 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0579a65d-dfff-4959-9634-12c9199c2e0e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:11.587282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0579a65d-dfff-4959-9634-12c9199c2e0e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:11.587598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0579a65d-dfff-4959-9634-12c9199c2e0e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:11.618867 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 118/221] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:11.666509 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:11.675815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0579a65d-dfff-4959-9634-12c9199c2e0e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:11.686285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0579a65d-dfff-4959-9634-12c9199c2e0e None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:11.764705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb1d63b-a6a6-4d48-890d-b293d29af5da None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:11.791897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 104/222] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 03:26:11.797751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0579a65d-dfff-4959-9634-12c9199c2e0e None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:11.840842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:11.896433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb1d63b-a6a6-4d48-890d-b293d29af5da None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:11.914005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:11.928070 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0579a65d-dfff-4959-9634-12c9199c2e0e None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:11.935003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a8d7ff-53ee-4daf-b13d-bdbc4085e33b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:11.935644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a8d7ff-53ee-4daf-b13d-bdbc4085e33b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:11.935923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a8d7ff-53ee-4daf-b13d-bdbc4085e33b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:11.936191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a8d7ff-53ee-4daf-b13d-bdbc4085e33b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:11.948230 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a8d7ff-53ee-4daf-b13d-bdbc4085e33b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-22a8d7ff-53ee-4daf-b13d-bdbc4085e33b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:11.953563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:11.954462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:11.955250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 119/223] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:26:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:11.955898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a8d7ff-53ee-4daf-b13d-bdbc4085e33b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:11.971244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad455cd-95cd-4977-a08e-62f18ab6998e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:11.972101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad455cd-95cd-4977-a08e-62f18ab6998e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:11.972578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad455cd-95cd-4977-a08e-62f18ab6998e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:11.972986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad455cd-95cd-4977-a08e-62f18ab6998e None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:11.974421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad455cd-95cd-4977-a08e-62f18ab6998e None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:11.976068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad455cd-95cd-4977-a08e-62f18ab6998e None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:11.978133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 105/224] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:11 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:11.983495 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad455cd-95cd-4977-a08e-62f18ab6998e None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:12.069264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a8d7ff-53ee-4daf-b13d-bdbc4085e33b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:12.092658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad455cd-95cd-4977-a08e-62f18ab6998e None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:12.093475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:12.112080 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16a2b8a-d241-4d53-8295-c78b532adb3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:12.112728 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16a2b8a-d241-4d53-8295-c78b532adb3f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:12.113006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16a2b8a-d241-4d53-8295-c78b532adb3f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:12.113861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16a2b8a-d241-4d53-8295-c78b532adb3f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:12.124680 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16a2b8a-d241-4d53-8295-c78b532adb3f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-f16a2b8a-d241-4d53-8295-c78b532adb3f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:12.129853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:12.131171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:12.131967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 120/225] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:26:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:12.132740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16a2b8a-d241-4d53-8295-c78b532adb3f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:12.135452 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2469ebf2-2ea8-42d3-8ad4-dc269f3d9009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:12.136561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2469ebf2-2ea8-42d3-8ad4-dc269f3d9009 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:12.137034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2469ebf2-2ea8-42d3-8ad4-dc269f3d9009 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:12.137496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2469ebf2-2ea8-42d3-8ad4-dc269f3d9009 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:12.138409 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2469ebf2-2ea8-42d3-8ad4-dc269f3d9009 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:12.138922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2469ebf2-2ea8-42d3-8ad4-dc269f3d9009 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:12.140499 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2469ebf2-2ea8-42d3-8ad4-dc269f3d9009 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:12.150185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 106/226] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:12.151278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2469ebf2-2ea8-42d3-8ad4-dc269f3d9009 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:12.176754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:12.239478 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16a2b8a-d241-4d53-8295-c78b532adb3f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:12.259623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2469ebf2-2ea8-42d3-8ad4-dc269f3d9009 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:12.276507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa0205f-3321-4284-ba84-55544a4701de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:12.277118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa0205f-3321-4284-ba84-55544a4701de None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:12.277397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa0205f-3321-4284-ba84-55544a4701de None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:12.277664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa0205f-3321-4284-ba84-55544a4701de None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:12.278427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa0205f-3321-4284-ba84-55544a4701de None admin] RBAC: Adding query filter params (role.id=9912acfb698e41ebbce6a5147fafe896, scope.project.id=5c50437619304d0d857f65feed1d19e3, user.id=3c47c2755dfc48c38c37c96e25db3829) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:12.278734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa0205f-3321-4284-ba84-55544a4701de None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:12.280195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa0205f-3321-4284-ba84-55544a4701de None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:12.286032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 121/227] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:12 2022] GET /identity/v3/role_assignments?user.id=3c47c2755dfc48c38c37c96e25db3829&scope.project.id=5c50437619304d0d857f65feed1d19e3&role.id=9912acfb698e41ebbce6a5147fafe896 => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:12.286892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa0205f-3321-4284-ba84-55544a4701de None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:12.299697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06afc08-90cc-4978-9185-a23f2a96795c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:12.300956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06afc08-90cc-4978-9185-a23f2a96795c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:12.301605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06afc08-90cc-4978-9185-a23f2a96795c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:12.302266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06afc08-90cc-4978-9185-a23f2a96795c None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:12.303262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06afc08-90cc-4978-9185-a23f2a96795c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:12.303765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06afc08-90cc-4978-9185-a23f2a96795c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:12.305510 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06afc08-90cc-4978-9185-a23f2a96795c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:12.319370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 107/228] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:12.324984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06afc08-90cc-4978-9185-a23f2a96795c None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:12.342187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:12.398197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa0205f-3321-4284-ba84-55544a4701de None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:12.438850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abbfafc-0cdb-409d-91f9-d0649b186717 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:12.439492 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abbfafc-0cdb-409d-91f9-d0649b186717 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:12.439776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abbfafc-0cdb-409d-91f9-d0649b186717 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:12.440151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abbfafc-0cdb-409d-91f9-d0649b186717 None admin] PATH_INFO: `/v3/users/1a8653bb588847d6a601967d2bae7a36` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:12.441940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abbfafc-0cdb-409d-91f9-d0649b186717 None admin] RBAC: Authorizing `identity:get_user(user_id=1a8653bb588847d6a601967d2bae7a36)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:12.443598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abbfafc-0cdb-409d-91f9-d0649b186717 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:12.445510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 122/229] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:12 2022] GET /identity/v3/users/1a8653bb588847d6a601967d2bae7a36 => generated 283 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:12.455365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abbfafc-0cdb-409d-91f9-d0649b186717 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:12.494555 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:12.530333 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06afc08-90cc-4978-9185-a23f2a96795c None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:12.575712 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0673c2c0-b494-4fb8-8a21-5f0f77cf5b69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:12.577436 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0673c2c0-b494-4fb8-8a21-5f0f77cf5b69 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:12.577855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0673c2c0-b494-4fb8-8a21-5f0f77cf5b69 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:12.578426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0673c2c0-b494-4fb8-8a21-5f0f77cf5b69 None admin] PATH_INFO: `/v3/users/1a8653bb588847d6a601967d2bae7a36` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:12.580145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0673c2c0-b494-4fb8-8a21-5f0f77cf5b69 None admin] RBAC: Authorizing `identity:get_user(user_id=1a8653bb588847d6a601967d2bae7a36)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:12.582607 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0673c2c0-b494-4fb8-8a21-5f0f77cf5b69 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:12.584898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 108/230] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:12 2022] GET /identity/v3/users/1a8653bb588847d6a601967d2bae7a36 => generated 283 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:12.589900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0673c2c0-b494-4fb8-8a21-5f0f77cf5b69 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:12.630482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abbfafc-0cdb-409d-91f9-d0649b186717 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:12.652175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:12.701148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf81cc5-94d0-49e3-8289-c2101b4f0b23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:12.701710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf81cc5-94d0-49e3-8289-c2101b4f0b23 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:12.701989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf81cc5-94d0-49e3-8289-c2101b4f0b23 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:12.702286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf81cc5-94d0-49e3-8289-c2101b4f0b23 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:12.703470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf81cc5-94d0-49e3-8289-c2101b4f0b23 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:12.706003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf81cc5-94d0-49e3-8289-c2101b4f0b23 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:12.711119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 123/231] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:12 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:12.715201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf81cc5-94d0-49e3-8289-c2101b4f0b23 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:12.785458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0673c2c0-b494-4fb8-8a21-5f0f77cf5b69 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:12.869033 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7823b113-3234-479b-ac20-c6c7b0b74186 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:12.870033 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7823b113-3234-479b-ac20-c6c7b0b74186 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:12.870535 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7823b113-3234-479b-ac20-c6c7b0b74186 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:12.871002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7823b113-3234-479b-ac20-c6c7b0b74186 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:12.874056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7823b113-3234-479b-ac20-c6c7b0b74186 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:12.875949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7823b113-3234-479b-ac20-c6c7b0b74186 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:12.880503 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 109/232] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:12 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:12.883966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf81cc5-94d0-49e3-8289-c2101b4f0b23 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:12.886756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7823b113-3234-479b-ac20-c6c7b0b74186 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:12.922771 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:12.929255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab88cd01-2907-4344-96c6-38bcc2fe9c22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:12.930230 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab88cd01-2907-4344-96c6-38bcc2fe9c22 None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:12.934730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab88cd01-2907-4344-96c6-38bcc2fe9c22 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:12.935254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab88cd01-2907-4344-96c6-38bcc2fe9c22 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3/users/1a8653bb588847d6a601967d2bae7a36/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:12.938942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab88cd01-2907-4344-96c6-38bcc2fe9c22 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c50437619304d0d857f65feed1d19e3, user_id=1a8653bb588847d6a601967d2bae7a36, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:12.943895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab88cd01-2907-4344-96c6-38bcc2fe9c22 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:13.030032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7823b113-3234-479b-ac20-c6c7b0b74186 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:13.041670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 124/233] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:26:12 2022] PUT /identity/v3/projects/5c50437619304d0d857f65feed1d19e3/users/1a8653bb588847d6a601967d2bae7a36/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:13.075887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242249f7-bdf4-4c65-880f-a8a1359baaa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:13.077079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242249f7-bdf4-4c65-880f-a8a1359baaa1 None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:13.077526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242249f7-bdf4-4c65-880f-a8a1359baaa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:13.077811 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242249f7-bdf4-4c65-880f-a8a1359baaa1 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3/users/1a8653bb588847d6a601967d2bae7a36/roles/d02a8e6c49b74aaa9403f8bd535b90bf` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:13.080382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242249f7-bdf4-4c65-880f-a8a1359baaa1 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c50437619304d0d857f65feed1d19e3, user_id=1a8653bb588847d6a601967d2bae7a36, role_id=d02a8e6c49b74aaa9403f8bd535b90bf)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:13.082029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242249f7-bdf4-4c65-880f-a8a1359baaa1 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:13.182992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 110/234] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:26:12 2022] PUT /identity/v3/projects/5c50437619304d0d857f65feed1d19e3/users/1a8653bb588847d6a601967d2bae7a36/roles/d02a8e6c49b74aaa9403f8bd535b90bf => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:13.745141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3444bfd9-ba81-4d04-a019-6135bcd580a3 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:13.746121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3444bfd9-ba81-4d04-a019-6135bcd580a3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:13.746666 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3444bfd9-ba81-4d04-a019-6135bcd580a3 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:13.749202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 125/235] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:13.755189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f47260d-d93c-4f96-a0d0-2208fbdeb046 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:13.756705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f47260d-d93c-4f96-a0d0-2208fbdeb046 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:13.757408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f47260d-d93c-4f96-a0d0-2208fbdeb046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:13.788479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-4f47260d-d93c-4f96-a0d0-2208fbdeb046 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:13.790022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4f47260d-d93c-4f96-a0d0-2208fbdeb046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:13.897688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 111/236] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:13.945397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d90f6ba-c45f-4f18-972e-15920c826b79 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:13.945699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d90f6ba-c45f-4f18-972e-15920c826b79 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:13.945952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d90f6ba-c45f-4f18-972e-15920c826b79 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:13.948285 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 126/237] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:13.953924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25207dd6-1706-4e0c-90de-17ba6e034bbd None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:13.954225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25207dd6-1706-4e0c-90de-17ba6e034bbd None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:13.954495 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25207dd6-1706-4e0c-90de-17ba6e034bbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:13.971141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:13.978616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-25207dd6-1706-4e0c-90de-17ba6e034bbd None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:13.979795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-25207dd6-1706-4e0c-90de-17ba6e034bbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:14.078930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 112/238] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:14.351659 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d90f6ba-c45f-4f18-972e-15920c826b79 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:14.405130 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:14.452169 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25207dd6-1706-4e0c-90de-17ba6e034bbd None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:14.508741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d90f6ba-c45f-4f18-972e-15920c826b79 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:14.577194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a146394c-86dd-45c1-a982-0166945b8837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:14.578178 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a146394c-86dd-45c1-a982-0166945b8837 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:14.578631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a146394c-86dd-45c1-a982-0166945b8837 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:14.579071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a146394c-86dd-45c1-a982-0166945b8837 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:14.601378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a146394c-86dd-45c1-a982-0166945b8837 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-a146394c-86dd-45c1-a982-0166945b8837 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:14.607158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:14.608003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:14.608786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:14.608786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:14.608786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:14.608786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:14.608786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:14.608786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:14.608786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:14.608786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:14.608786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:14.608786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:14.608786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:14.608786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:14.614118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 127/239] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:14 2022] GET /identity/v3/roles/reader => generated 84 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:14.619692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a146394c-86dd-45c1-a982-0166945b8837 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:14.634616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25207dd6-1706-4e0c-90de-17ba6e034bbd None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:14.675660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a10bb8d-c1fe-4b13-bcbf-82e66842beba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:14.676875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a10bb8d-c1fe-4b13-bcbf-82e66842beba None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:14.677325 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a10bb8d-c1fe-4b13-bcbf-82e66842beba None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:14.677761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a10bb8d-c1fe-4b13-bcbf-82e66842beba None admin] PATH_INFO: `/v3/domains/default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:14.679406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a10bb8d-c1fe-4b13-bcbf-82e66842beba None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:14.681301 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a10bb8d-c1fe-4b13-bcbf-82e66842beba None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:14.683398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 113/240] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:14.693689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a10bb8d-c1fe-4b13-bcbf-82e66842beba None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:14.702526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:14.774358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:14.778044 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a146394c-86dd-45c1-a982-0166945b8837 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:14.814454 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd253cd7-c2cf-4b29-b9d5-103f8508a0d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:14.815351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd253cd7-c2cf-4b29-b9d5-103f8508a0d4 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:14.815831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd253cd7-c2cf-4b29-b9d5-103f8508a0d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:14.816274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd253cd7-c2cf-4b29-b9d5-103f8508a0d4 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:14.817166 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd253cd7-c2cf-4b29-b9d5-103f8508a0d4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:14.817606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd253cd7-c2cf-4b29-b9d5-103f8508a0d4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:14.819209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd253cd7-c2cf-4b29-b9d5-103f8508a0d4 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:14.824972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a10bb8d-c1fe-4b13-bcbf-82e66842beba None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:14.828884 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 128/241] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:14 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:14.829376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32e2847-29ce-4ae9-9bdf-9dfe07ac03b7 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:14.829779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32e2847-29ce-4ae9-9bdf-9dfe07ac03b7 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:14.830062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32e2847-29ce-4ae9-9bdf-9dfe07ac03b7 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:14.831289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 129/242] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:14.834048 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32e2847-29ce-4ae9-9bdf-9dfe07ac03b7 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:14.863619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c528aee-e3a5-49a4-92c6-f730147b41c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:14.864601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c528aee-e3a5-49a4-92c6-f730147b41c1 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:14.865017 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c528aee-e3a5-49a4-92c6-f730147b41c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:14.865426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c528aee-e3a5-49a4-92c6-f730147b41c1 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:14.866271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c528aee-e3a5-49a4-92c6-f730147b41c1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:14.867873 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c528aee-e3a5-49a4-92c6-f730147b41c1 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:14.946443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32e2847-29ce-4ae9-9bdf-9dfe07ac03b7 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:14.958870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 114/243] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:26:14 2022] POST /identity/v3/users => generated 293 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:26:14.960737 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6932bc60-4ac9-4e45-ba3e-f64e6806ea39 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:14.961284 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6932bc60-4ac9-4e45-ba3e-f64e6806ea39 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:14.961792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6932bc60-4ac9-4e45-ba3e-f64e6806ea39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:14.986305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-6932bc60-4ac9-4e45-ba3e-f64e6806ea39 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:14.988725 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6932bc60-4ac9-4e45-ba3e-f64e6806ea39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:15.003724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ceb970-31ed-436e-b5de-77a7142a5901 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:15.005238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ceb970-31ed-436e-b5de-77a7142a5901 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:15.005389 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ceb970-31ed-436e-b5de-77a7142a5901 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:15.005578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ceb970-31ed-436e-b5de-77a7142a5901 None admin] PATH_INFO: `/v3/users/f59235a684414536a3d259252cb5b70d` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:15.030995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ceb970-31ed-436e-b5de-77a7142a5901 None admin] RBAC: Authorizing `identity:get_user(user_id=f59235a684414536a3d259252cb5b70d)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:15.032581 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ceb970-31ed-436e-b5de-77a7142a5901 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:15.035473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 130/244] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:14 2022] GET /identity/v3/users/f59235a684414536a3d259252cb5b70d => generated 297 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:15.040309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ceb970-31ed-436e-b5de-77a7142a5901 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:15.064906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:15.077000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:15.084321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 115/245] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:15.193951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ceb970-31ed-436e-b5de-77a7142a5901 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:15.241107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b435c1-f732-4737-9a14-49a0e74ebe72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:15.241768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b435c1-f732-4737-9a14-49a0e74ebe72 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:15.242045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b435c1-f732-4737-9a14-49a0e74ebe72 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:15.242373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b435c1-f732-4737-9a14-49a0e74ebe72 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:15.319612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b435c1-f732-4737-9a14-49a0e74ebe72 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:15.321134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b435c1-f732-4737-9a14-49a0e74ebe72 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:15.342672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:15.420318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6932bc60-4ac9-4e45-ba3e-f64e6806ea39 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:15.439564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 131/246] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:15.447263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b435c1-f732-4737-9a14-49a0e74ebe72 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:15.550968 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6932bc60-4ac9-4e45-ba3e-f64e6806ea39 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:15.581561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b435c1-f732-4737-9a14-49a0e74ebe72 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:15.589916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8a4321-8a74-4f50-9e88-71b860f41a37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:15.590598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8a4321-8a74-4f50-9e88-71b860f41a37 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:15.590874 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8a4321-8a74-4f50-9e88-71b860f41a37 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:15.591144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8a4321-8a74-4f50-9e88-71b860f41a37 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:15.607534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:15.608476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8a4321-8a74-4f50-9e88-71b860f41a37 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-fe8a4321-8a74-4f50-9e88-71b860f41a37 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:15.613501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:15.614556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:15.615334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 116/247] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:15.616934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f9c942-2a6b-419b-8272-3aadfe36efa4 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:15.617184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f9c942-2a6b-419b-8272-3aadfe36efa4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:15.617473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f9c942-2a6b-419b-8272-3aadfe36efa4 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:15.619710 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 117/248] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:15.623443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f9c942-2a6b-419b-8272-3aadfe36efa4 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:15.633069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9b8a19-7f31-42e1-92aa-04d71de42e2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:15.634187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9b8a19-7f31-42e1-92aa-04d71de42e2d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:15.634333 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9b8a19-7f31-42e1-92aa-04d71de42e2d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:15.634593 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9b8a19-7f31-42e1-92aa-04d71de42e2d None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:15.635745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9b8a19-7f31-42e1-92aa-04d71de42e2d None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:15.637201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9b8a19-7f31-42e1-92aa-04d71de42e2d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:15.638854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 132/249] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:15 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:15.640171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8555fd1-7840-4d4c-bad8-6601d2df8081 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:15.640531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8555fd1-7840-4d4c-bad8-6601d2df8081 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:15.640727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8555fd1-7840-4d4c-bad8-6601d2df8081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:15.661723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-b8555fd1-7840-4d4c-bad8-6601d2df8081 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:15.662967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b8555fd1-7840-4d4c-bad8-6601d2df8081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:15.683210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:15.766331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 133/250] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:15.767200 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f9c942-2a6b-419b-8272-3aadfe36efa4 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:15.770444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8555fd1-7840-4d4c-bad8-6601d2df8081 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:15.830365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e5647b-83d5-47dc-9baf-0aebc8a71199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:15.831616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e5647b-83d5-47dc-9baf-0aebc8a71199 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:15.832567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e5647b-83d5-47dc-9baf-0aebc8a71199 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:15.832977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e5647b-83d5-47dc-9baf-0aebc8a71199 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:15.833816 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e5647b-83d5-47dc-9baf-0aebc8a71199 None admin] RBAC: Adding query filter params (name=member) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:15.834292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e5647b-83d5-47dc-9baf-0aebc8a71199 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:15.840419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e5647b-83d5-47dc-9baf-0aebc8a71199 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:15.858265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 118/251] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:15.860695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e5647b-83d5-47dc-9baf-0aebc8a71199 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:15.939580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8555fd1-7840-4d4c-bad8-6601d2df8081 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:15.963350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:15.989402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e5647b-83d5-47dc-9baf-0aebc8a71199 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:15.993395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c96db8-29d4-4162-8221-e08ec84904ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:15.994260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c96db8-29d4-4162-8221-e08ec84904ab None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:15.994676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c96db8-29d4-4162-8221-e08ec84904ab None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:15.995994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c96db8-29d4-4162-8221-e08ec84904ab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:15.996868 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c96db8-29d4-4162-8221-e08ec84904ab None admin] RBAC: Adding query filter params (role.id=74e1fd5acabf47a7b788b2dc78d1fb3e, scope.project.id=5c50437619304d0d857f65feed1d19e3, user.id=f59235a684414536a3d259252cb5b70d) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:15.999890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c96db8-29d4-4162-8221-e08ec84904ab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:16.007830 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c96db8-29d4-4162-8221-e08ec84904ab None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:16.010122 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:16.020790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 134/252] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:15 2022] GET /identity/v3/role_assignments?user.id=f59235a684414536a3d259252cb5b70d&scope.project.id=5c50437619304d0d857f65feed1d19e3&role.id=74e1fd5acabf47a7b788b2dc78d1fb3e => generated 259 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:16.020790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd46729e-c3cf-49b8-8428-2408ee70bf35 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:16.020790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd46729e-c3cf-49b8-8428-2408ee70bf35 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:16.020790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd46729e-c3cf-49b8-8428-2408ee70bf35 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:16.021163 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 135/253] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:16.026841 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce7ed13-7415-4e5e-a84b-af8a237c02bc None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:16.027127 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce7ed13-7415-4e5e-a84b-af8a237c02bc None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:16.027404 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce7ed13-7415-4e5e-a84b-af8a237c02bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:16.040157 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f14df5a-7ed3-482e-bae3-06f4003f2ecc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:16.041065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f14df5a-7ed3-482e-bae3-06f4003f2ecc None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:16.044571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f14df5a-7ed3-482e-bae3-06f4003f2ecc None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:16.044997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f14df5a-7ed3-482e-bae3-06f4003f2ecc None admin] PATH_INFO: `/v3/users/3c47c2755dfc48c38c37c96e25db3829` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:16.048532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f14df5a-7ed3-482e-bae3-06f4003f2ecc None admin] RBAC: Authorizing `identity:get_user(user_id=3c47c2755dfc48c38c37c96e25db3829)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:16.050311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f14df5a-7ed3-482e-bae3-06f4003f2ecc None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:16.052363 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 119/254] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:15 2022] GET /identity/v3/users/3c47c2755dfc48c38c37c96e25db3829 => generated 297 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:16.059643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f14df5a-7ed3-482e-bae3-06f4003f2ecc None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:16.069149 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-cce7ed13-7415-4e5e-a84b-af8a237c02bc None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:16.073403 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-cce7ed13-7415-4e5e-a84b-af8a237c02bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:16.212744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 136/255] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:26:16.213647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce7ed13-7415-4e5e-a84b-af8a237c02bc None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:16.297337 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f14df5a-7ed3-482e-bae3-06f4003f2ecc None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:16.352864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331f85e9-f58f-4689-a056-986d9b492a37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:16.354934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331f85e9-f58f-4689-a056-986d9b492a37 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:16.355215 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331f85e9-f58f-4689-a056-986d9b492a37 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:16.355485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331f85e9-f58f-4689-a056-986d9b492a37 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:16.356775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331f85e9-f58f-4689-a056-986d9b492a37 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:16.359401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331f85e9-f58f-4689-a056-986d9b492a37 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:16.360979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 120/256] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:16 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:16.367327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331f85e9-f58f-4689-a056-986d9b492a37 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:16.433009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce7ed13-7415-4e5e-a84b-af8a237c02bc None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:16.452578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:16.487571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:16.491930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f493d8ad-c014-4e59-bebc-33d5b59f420b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:16.492573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f493d8ad-c014-4e59-bebc-33d5b59f420b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:16.492851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f493d8ad-c014-4e59-bebc-33d5b59f420b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:16.493122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f493d8ad-c014-4e59-bebc-33d5b59f420b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:16.517849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f493d8ad-c014-4e59-bebc-33d5b59f420b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-f493d8ad-c014-4e59-bebc-33d5b59f420b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:16.524419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:16.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:16.526081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 137/257] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:26:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:16.536627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f493d8ad-c014-4e59-bebc-33d5b59f420b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:16.592335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331f85e9-f58f-4689-a056-986d9b492a37 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:16.632785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d615e6-d266-4d56-96bf-07390e723529 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:16.633459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d615e6-d266-4d56-96bf-07390e723529 None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:16.633739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d615e6-d266-4d56-96bf-07390e723529 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:16.634011 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d615e6-d266-4d56-96bf-07390e723529 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3/users/3c47c2755dfc48c38c37c96e25db3829/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:16.636016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d615e6-d266-4d56-96bf-07390e723529 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c50437619304d0d857f65feed1d19e3, user_id=3c47c2755dfc48c38c37c96e25db3829, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:16.638061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d615e6-d266-4d56-96bf-07390e723529 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:16.728246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f493d8ad-c014-4e59-bebc-33d5b59f420b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:16.763833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07283c61-c5a1-49da-a47d-c08a33edc5bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:16.764565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07283c61-c5a1-49da-a47d-c08a33edc5bd None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:16.764846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07283c61-c5a1-49da-a47d-c08a33edc5bd None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:16.765116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07283c61-c5a1-49da-a47d-c08a33edc5bd None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:16.765847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07283c61-c5a1-49da-a47d-c08a33edc5bd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:16.766170 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07283c61-c5a1-49da-a47d-c08a33edc5bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:16.767586 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07283c61-c5a1-49da-a47d-c08a33edc5bd None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:16.780753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 138/258] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:16.781644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07283c61-c5a1-49da-a47d-c08a33edc5bd None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:16.787282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 121/259] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:26:16 2022] PUT /identity/v3/projects/5c50437619304d0d857f65feed1d19e3/users/3c47c2755dfc48c38c37c96e25db3829/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:16.796989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d615e6-d266-4d56-96bf-07390e723529 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:16.832398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:16.905447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07283c61-c5a1-49da-a47d-c08a33edc5bd None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:16.943713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0890eff4-bab6-48dd-95e7-8fc7b5185218 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:16.944576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0890eff4-bab6-48dd-95e7-8fc7b5185218 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:16.944922 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0890eff4-bab6-48dd-95e7-8fc7b5185218 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:16.945259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0890eff4-bab6-48dd-95e7-8fc7b5185218 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:16.957691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0890eff4-bab6-48dd-95e7-8fc7b5185218 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-0890eff4-bab6-48dd-95e7-8fc7b5185218 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:16.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:16.965297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:16.967264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 139/260] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:26:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:16.974192 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0890eff4-bab6-48dd-95e7-8fc7b5185218 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:16.985154 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d615e6-d266-4d56-96bf-07390e723529 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:17.008479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:17.082925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2794948-ec88-489a-95a4-3cd0f5e346e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:17.083589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2794948-ec88-489a-95a4-3cd0f5e346e4 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:17.083873 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2794948-ec88-489a-95a4-3cd0f5e346e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:17.084152 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2794948-ec88-489a-95a4-3cd0f5e346e4 None admin] PATH_INFO: `/v3/users/1a8653bb588847d6a601967d2bae7a36` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:17.090926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2794948-ec88-489a-95a4-3cd0f5e346e4 None admin] RBAC: Authorizing `identity:get_user(user_id=1a8653bb588847d6a601967d2bae7a36)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:17.092404 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2794948-ec88-489a-95a4-3cd0f5e346e4 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:17.094272 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 122/261] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:16 2022] GET /identity/v3/users/1a8653bb588847d6a601967d2bae7a36 => generated 283 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:17.114606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:17.118345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2794948-ec88-489a-95a4-3cd0f5e346e4 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:17.122943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0890eff4-bab6-48dd-95e7-8fc7b5185218 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:17.163321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838f0d3c-5f18-45fd-b4f9-46e22f5f8525 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:17.167416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838f0d3c-5f18-45fd-b4f9-46e22f5f8525 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:17.167697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838f0d3c-5f18-45fd-b4f9-46e22f5f8525 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:17.167967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838f0d3c-5f18-45fd-b4f9-46e22f5f8525 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:17.168642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838f0d3c-5f18-45fd-b4f9-46e22f5f8525 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:17.168943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838f0d3c-5f18-45fd-b4f9-46e22f5f8525 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:17.170370 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838f0d3c-5f18-45fd-b4f9-46e22f5f8525 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:17.195711 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 140/262] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:17.211807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838f0d3c-5f18-45fd-b4f9-46e22f5f8525 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:17.368284 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838f0d3c-5f18-45fd-b4f9-46e22f5f8525 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:17.423987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a82b84-73d4-4cc8-a5e4-99485550fe26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:17.424631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a82b84-73d4-4cc8-a5e4-99485550fe26 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:17.424964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a82b84-73d4-4cc8-a5e4-99485550fe26 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:17.425235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a82b84-73d4-4cc8-a5e4-99485550fe26 None admin] PATH_INFO: `/v3/users/1a8653bb588847d6a601967d2bae7a36` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:17.430912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a82b84-73d4-4cc8-a5e4-99485550fe26 None admin] RBAC: Authorizing `identity:get_user(user_id=1a8653bb588847d6a601967d2bae7a36)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:17.432684 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a82b84-73d4-4cc8-a5e4-99485550fe26 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:17.434942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 141/263] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:17 2022] GET /identity/v3/users/1a8653bb588847d6a601967d2bae7a36 => generated 283 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:17.440990 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a82b84-73d4-4cc8-a5e4-99485550fe26 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:17.456406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2794948-ec88-489a-95a4-3cd0f5e346e4 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:17.494578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:17.526886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce9c2f2-63ad-44df-9d38-7d3d91eb1438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:17.527519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce9c2f2-63ad-44df-9d38-7d3d91eb1438 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:17.527800 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce9c2f2-63ad-44df-9d38-7d3d91eb1438 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:17.528074 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce9c2f2-63ad-44df-9d38-7d3d91eb1438 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:17.580827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a82b84-73d4-4cc8-a5e4-99485550fe26 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:17.607771 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:17.618303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8ac16d-a47c-4110-a4ca-5cf980589c4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:17.622276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8ac16d-a47c-4110-a4ca-5cf980589c4a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:17.622276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8ac16d-a47c-4110-a4ca-5cf980589c4a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:17.622276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8ac16d-a47c-4110-a4ca-5cf980589c4a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:17.649662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce9c2f2-63ad-44df-9d38-7d3d91eb1438 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:17.651170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce9c2f2-63ad-44df-9d38-7d3d91eb1438 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:17.693360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8ac16d-a47c-4110-a4ca-5cf980589c4a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:17.694887 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8ac16d-a47c-4110-a4ca-5cf980589c4a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:17.725070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:17.801230 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 142/264] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 03:26:17.803549 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e960c58-090a-4af5-842e-57e7e92a7a4c None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:17.803843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e960c58-090a-4af5-842e-57e7e92a7a4c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:17.804114 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e960c58-090a-4af5-842e-57e7e92a7a4c None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:17.805411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 143/265] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:17.807509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e960c58-090a-4af5-842e-57e7e92a7a4c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:17.823882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 123/266] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 706 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:17.825240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dac8a0-331a-46e1-9d93-73ed82e7ca29 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:17.825528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dac8a0-331a-46e1-9d93-73ed82e7ca29 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:17.825798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dac8a0-331a-46e1-9d93-73ed82e7ca29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:17.849255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f9dac8a0-331a-46e1-9d93-73ed82e7ca29 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:17.850501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f9dac8a0-331a-46e1-9d93-73ed82e7ca29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:17.962801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 124/267] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:17.963413 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dac8a0-331a-46e1-9d93-73ed82e7ca29 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:17.972439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:18.064441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e960c58-090a-4af5-842e-57e7e92a7a4c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:18.082712 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dac8a0-331a-46e1-9d93-73ed82e7ca29 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:18.102237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:18.127028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4c936a-76f6-4944-87b4-f4c90dd7474a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:18.127658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4c936a-76f6-4944-87b4-f4c90dd7474a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:18.127930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4c936a-76f6-4944-87b4-f4c90dd7474a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:18.128195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4c936a-76f6-4944-87b4-f4c90dd7474a None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:18.129956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4c936a-76f6-4944-87b4-f4c90dd7474a None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:18.131729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4c936a-76f6-4944-87b4-f4c90dd7474a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:18.138375 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbc1030-6e7b-4d7a-bd17-f8e576b996d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:18.139019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbc1030-6e7b-4d7a-bd17-f8e576b996d1 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:18.139296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbc1030-6e7b-4d7a-bd17-f8e576b996d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:18.139565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbc1030-6e7b-4d7a-bd17-f8e576b996d1 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:18.140095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 125/268] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:17 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:18.140695 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbc1030-6e7b-4d7a-bd17-f8e576b996d1 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:18.142115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbc1030-6e7b-4d7a-bd17-f8e576b996d1 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:18.143776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 144/269] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:17 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:18.150878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4c936a-76f6-4944-87b4-f4c90dd7474a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:18.157604 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9986e2-2d7b-401c-ba77-a005fdc0d06b None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:18.157880 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9986e2-2d7b-401c-ba77-a005fdc0d06b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:18.158165 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9986e2-2d7b-401c-ba77-a005fdc0d06b None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:18.159551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 145/270] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:26:18.160311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9986e2-2d7b-401c-ba77-a005fdc0d06b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:18.258528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:18.284537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4c936a-76f6-4944-87b4-f4c90dd7474a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:18.290430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9986e2-2d7b-401c-ba77-a005fdc0d06b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:18.326952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dad7890-99ad-4c84-8408-cf91f0882b0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:18.327557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dad7890-99ad-4c84-8408-cf91f0882b0d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:18.327829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dad7890-99ad-4c84-8408-cf91f0882b0d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:18.328094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dad7890-99ad-4c84-8408-cf91f0882b0d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:18.328822 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dad7890-99ad-4c84-8408-cf91f0882b0d None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=5c50437619304d0d857f65feed1d19e3, user.id=1a8653bb588847d6a601967d2bae7a36) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:18.329125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dad7890-99ad-4c84-8408-cf91f0882b0d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:18.330596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dad7890-99ad-4c84-8408-cf91f0882b0d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:18.336687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 146/271] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:18 2022] GET /identity/v3/role_assignments?user.id=1a8653bb588847d6a601967d2bae7a36&scope.project.id=5c50437619304d0d857f65feed1d19e3&role.id=43ca25467bc6484da3a50508efb79230 => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:18.337917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a45ae5-c9eb-4fb4-b3ff-716d02293e0e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:18.338214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a45ae5-c9eb-4fb4-b3ff-716d02293e0e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:18.338482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a45ae5-c9eb-4fb4-b3ff-716d02293e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:18.339322 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f65be2a-36b7-4fab-80b4-4a96b5f26873 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:18.339942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f65be2a-36b7-4fab-80b4-4a96b5f26873 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:18.340218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f65be2a-36b7-4fab-80b4-4a96b5f26873 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:18.340485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f65be2a-36b7-4fab-80b4-4a96b5f26873 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:18.341335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f65be2a-36b7-4fab-80b4-4a96b5f26873 None admin] RBAC: Adding query filter params (role.id=d02a8e6c49b74aaa9403f8bd535b90bf, scope.project.id=5c50437619304d0d857f65feed1d19e3, user.id=1a8653bb588847d6a601967d2bae7a36) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:18.341639 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f65be2a-36b7-4fab-80b4-4a96b5f26873 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:18.349560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f65be2a-36b7-4fab-80b4-4a96b5f26873 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:18.354117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 126/272] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:18 2022] GET /identity/v3/role_assignments?user.id=1a8653bb588847d6a601967d2bae7a36&scope.project.id=5c50437619304d0d857f65feed1d19e3&role.id=d02a8e6c49b74aaa9403f8bd535b90bf => generated 608 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:18.354991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f65be2a-36b7-4fab-80b4-4a96b5f26873 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:18.362939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c9a45ae5-c9eb-4fb4-b3ff-716d02293e0e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:18.365187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c9a45ae5-c9eb-4fb4-b3ff-716d02293e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:18.431345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:18.452697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 147/273] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:18.581892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f65be2a-36b7-4fab-80b4-4a96b5f26873 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:18.625975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76dc27da-b1c8-4733-9fa6-5d3c587c588c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:18.626662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76dc27da-b1c8-4733-9fa6-5d3c587c588c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:18.626943 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76dc27da-b1c8-4733-9fa6-5d3c587c588c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:18.627217 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76dc27da-b1c8-4733-9fa6-5d3c587c588c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:18.628398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76dc27da-b1c8-4733-9fa6-5d3c587c588c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:18.630243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76dc27da-b1c8-4733-9fa6-5d3c587c588c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:18.632501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 127/274] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:18.638096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a45ae5-c9eb-4fb4-b3ff-716d02293e0e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:18.754051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a45ae5-c9eb-4fb4-b3ff-716d02293e0e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:18.792341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f913a2-6cec-40f7-9fc3-01d65f60e9ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:18.792954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f913a2-6cec-40f7-9fc3-01d65f60e9ff None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:18.793236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f913a2-6cec-40f7-9fc3-01d65f60e9ff None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:18.793507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f913a2-6cec-40f7-9fc3-01d65f60e9ff None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:18.794177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f913a2-6cec-40f7-9fc3-01d65f60e9ff None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:18.795655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f913a2-6cec-40f7-9fc3-01d65f60e9ff None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:18.905326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76dc27da-b1c8-4733-9fa6-5d3c587c588c None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:18.908501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 148/275] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:26:18 2022] POST /identity/v3/users => generated 293 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:26:18.917216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:19.027658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76dc27da-b1c8-4733-9fa6-5d3c587c588c None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:19.035119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db31fb4-dd5b-4324-8a5c-1c0c69bd7719 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:19.035879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db31fb4-dd5b-4324-8a5c-1c0c69bd7719 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:19.036230 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db31fb4-dd5b-4324-8a5c-1c0c69bd7719 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:19.037984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 149/276] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:26:19.043800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d428115b-9b60-4e01-bdfd-648ec21b835b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:19.044361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d428115b-9b60-4e01-bdfd-648ec21b835b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:19.044809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d428115b-9b60-4e01-bdfd-648ec21b835b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:19.064356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d428115b-9b60-4e01-bdfd-648ec21b835b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:19.065983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d428115b-9b60-4e01-bdfd-648ec21b835b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:19.070775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a574fb96-43bd-47e4-9b60-743d00a8e798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:19.071427 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a574fb96-43bd-47e4-9b60-743d00a8e798 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:19.071701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a574fb96-43bd-47e4-9b60-743d00a8e798 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:19.071967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a574fb96-43bd-47e4-9b60-743d00a8e798 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:19.085846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a574fb96-43bd-47e4-9b60-743d00a8e798 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-a574fb96-43bd-47e4-9b60-743d00a8e798 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:19.091214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:19.092091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:19.092878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 128/277] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:19.099136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a574fb96-43bd-47e4-9b60-743d00a8e798 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:19.113986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:19.162266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 150/278] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:26:19.186285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:19.218853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a574fb96-43bd-47e4-9b60-743d00a8e798 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:19.256915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a38571-b8f9-4a65-b09e-8d1a36ed71a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:19.257582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a38571-b8f9-4a65-b09e-8d1a36ed71a8 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:19.257866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a38571-b8f9-4a65-b09e-8d1a36ed71a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:19.258140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a38571-b8f9-4a65-b09e-8d1a36ed71a8 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:19.258834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a38571-b8f9-4a65-b09e-8d1a36ed71a8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:19.259140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a38571-b8f9-4a65-b09e-8d1a36ed71a8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:19.260563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a38571-b8f9-4a65-b09e-8d1a36ed71a8 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:19.270636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 129/279] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:19.276050 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d428115b-9b60-4e01-bdfd-648ec21b835b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:19.398286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d428115b-9b60-4e01-bdfd-648ec21b835b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:19.437842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9568df05-b3a9-4f0e-8310-9cc7f794c920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:19.438522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9568df05-b3a9-4f0e-8310-9cc7f794c920 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:19.438802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9568df05-b3a9-4f0e-8310-9cc7f794c920 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:19.439073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9568df05-b3a9-4f0e-8310-9cc7f794c920 None admin] PATH_INFO: `/v3/users/d2825a6523264021bf18ee593b51215e` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:19.449926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a38571-b8f9-4a65-b09e-8d1a36ed71a8 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:19.454361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9568df05-b3a9-4f0e-8310-9cc7f794c920 None admin] RBAC: Authorizing `identity:get_user(user_id=d2825a6523264021bf18ee593b51215e)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:19.455831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9568df05-b3a9-4f0e-8310-9cc7f794c920 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:19.458501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 151/280] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:19 2022] GET /identity/v3/users/d2825a6523264021bf18ee593b51215e => generated 293 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:19.464220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9568df05-b3a9-4f0e-8310-9cc7f794c920 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:19.501895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:19.584805 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9568df05-b3a9-4f0e-8310-9cc7f794c920 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:19.588358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a38571-b8f9-4a65-b09e-8d1a36ed71a8 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:19.619150 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:19.624648 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ebaef0-fd7c-493c-a3ea-8d6feb20c604 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:19.625310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ebaef0-fd7c-493c-a3ea-8d6feb20c604 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:19.625589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ebaef0-fd7c-493c-a3ea-8d6feb20c604 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:19.625867 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ebaef0-fd7c-493c-a3ea-8d6feb20c604 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:19.626614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ebaef0-fd7c-493c-a3ea-8d6feb20c604 None admin] RBAC: Adding query filter params (role.id=9912acfb698e41ebbce6a5147fafe896, user.id=d2825a6523264021bf18ee593b51215e, scope.system=all) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:19.626917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ebaef0-fd7c-493c-a3ea-8d6feb20c604 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:19.628786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ebaef0-fd7c-493c-a3ea-8d6feb20c604 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:19.629679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df186703-5cd2-4bbe-8da4-304bc4f38bce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:19.630539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df186703-5cd2-4bbe-8da4-304bc4f38bce None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:19.631222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df186703-5cd2-4bbe-8da4-304bc4f38bce None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:19.631703 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df186703-5cd2-4bbe-8da4-304bc4f38bce None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:19.635227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 152/281] 10.222.0.65 () {56 vars in 1308 bytes} [Wed Aug 24 03:26:19 2022] GET /identity/v3/role_assignments?user.id=d2825a6523264021bf18ee593b51215e&scope.system=all&role.id=9912acfb698e41ebbce6a5147fafe896 => generated 226 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:19.636425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9f82c9-3669-498b-95a5-814317c1f0f3 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:19.636705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9f82c9-3669-498b-95a5-814317c1f0f3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:19.636974 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9f82c9-3669-498b-95a5-814317c1f0f3 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:19.638185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 153/282] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:19 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:19.645531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df186703-5cd2-4bbe-8da4-304bc4f38bce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:19.647959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a269e2-5596-44d3-8f72-2ea4f79fba14 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:19.648247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a269e2-5596-44d3-8f72-2ea4f79fba14 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:19.648630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a269e2-5596-44d3-8f72-2ea4f79fba14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-df186703-5cd2-4bbe-8da4-304bc4f38bce None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:19.651741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:19.652695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:19.653479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:19.653479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:19.653479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:19.653479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:19.653479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:19.653479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:19.653479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:19.653479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:19.653479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:19.653479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:19.653479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:19.653479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:19.656237 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 130/283] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:19 2022] GET /identity/v3/roles/reader => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:19.662615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df186703-5cd2-4bbe-8da4-304bc4f38bce None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:19.687504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-24a269e2-5596-44d3-8f72-2ea4f79fba14 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:19.688743 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-24a269e2-5596-44d3-8f72-2ea4f79fba14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:19.780801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df186703-5cd2-4bbe-8da4-304bc4f38bce None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:19.816973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f52f0eb-2c2a-4dec-a71e-d9eb889fb894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:19.818236 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f52f0eb-2c2a-4dec-a71e-d9eb889fb894 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:19.818785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f52f0eb-2c2a-4dec-a71e-d9eb889fb894 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:19.819351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f52f0eb-2c2a-4dec-a71e-d9eb889fb894 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:19.820222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f52f0eb-2c2a-4dec-a71e-d9eb889fb894 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:19.820586 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 154/284] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:26:19.820932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f52f0eb-2c2a-4dec-a71e-d9eb889fb894 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:19.822656 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f52f0eb-2c2a-4dec-a71e-d9eb889fb894 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:19.832552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 131/285] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:19 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:19.839501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a269e2-5596-44d3-8f72-2ea4f79fba14 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:19.883921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:19.997362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a269e2-5596-44d3-8f72-2ea4f79fba14 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:20.033498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1e7670-9285-402d-99f3-a079e2dbd3e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:20.034119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1e7670-9285-402d-99f3-a079e2dbd3e7 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:20.034483 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1e7670-9285-402d-99f3-a079e2dbd3e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:20.035287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1e7670-9285-402d-99f3-a079e2dbd3e7 None admin] PATH_INFO: `/v3/users/f59235a684414536a3d259252cb5b70d` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:20.037001 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1e7670-9285-402d-99f3-a079e2dbd3e7 None admin] RBAC: Authorizing `identity:get_user(user_id=f59235a684414536a3d259252cb5b70d)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:20.038888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1e7670-9285-402d-99f3-a079e2dbd3e7 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:20.040753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 155/286] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:19 2022] GET /identity/v3/users/f59235a684414536a3d259252cb5b70d => generated 297 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:20.046502 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f52f0eb-2c2a-4dec-a71e-d9eb889fb894 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:20.102758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:20.150579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1e7670-9285-402d-99f3-a079e2dbd3e7 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:20.166652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f52f0eb-2c2a-4dec-a71e-d9eb889fb894 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:20.281765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71dde22-1bcc-46e0-8e0d-695215a9c7a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:20.282500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71dde22-1bcc-46e0-8e0d-695215a9c7a9 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:20.282791 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71dde22-1bcc-46e0-8e0d-695215a9c7a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:20.283141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71dde22-1bcc-46e0-8e0d-695215a9c7a9 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:20.284279 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71dde22-1bcc-46e0-8e0d-695215a9c7a9 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:20.286823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71dde22-1bcc-46e0-8e0d-695215a9c7a9 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:20.287517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 132/287] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:20 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:20.295143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71dde22-1bcc-46e0-8e0d-695215a9c7a9 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:20.304310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:20.321922 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1e7670-9285-402d-99f3-a079e2dbd3e7 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:20.424910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a96e699-d387-4ac0-b0c7-01d3048a956e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:20.425910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a96e699-d387-4ac0-b0c7-01d3048a956e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:20.428221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a96e699-d387-4ac0-b0c7-01d3048a956e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:20.428756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a96e699-d387-4ac0-b0c7-01d3048a956e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:20.452470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a96e699-d387-4ac0-b0c7-01d3048a956e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-7a96e699-d387-4ac0-b0c7-01d3048a956e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:20.461605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:20.462536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:20.463342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:20.463342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:20.463342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:20.463342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:20.463342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:20.463342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:20.463342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:20.463342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:20.463342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:20.463342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:20.463342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:20.463342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:20.476129 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 156/288] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:20.477353 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a96e699-d387-4ac0-b0c7-01d3048a956e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:20.549808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71dde22-1bcc-46e0-8e0d-695215a9c7a9 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:20.572799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:20.609446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831aa1cb-20a6-455b-9c4a-61b821a47233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:20.617870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831aa1cb-20a6-455b-9c4a-61b821a47233 None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:20.618357 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831aa1cb-20a6-455b-9c4a-61b821a47233 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:20.618770 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831aa1cb-20a6-455b-9c4a-61b821a47233 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3/users/f59235a684414536a3d259252cb5b70d/roles/74e1fd5acabf47a7b788b2dc78d1fb3e` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:20.624809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831aa1cb-20a6-455b-9c4a-61b821a47233 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c50437619304d0d857f65feed1d19e3, user_id=f59235a684414536a3d259252cb5b70d, role_id=74e1fd5acabf47a7b788b2dc78d1fb3e)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:20.630116 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831aa1cb-20a6-455b-9c4a-61b821a47233 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:20.710190 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a96e699-d387-4ac0-b0c7-01d3048a956e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:20.721707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 133/289] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:26:20 2022] PUT /identity/v3/projects/5c50437619304d0d857f65feed1d19e3/users/f59235a684414536a3d259252cb5b70d/roles/74e1fd5acabf47a7b788b2dc78d1fb3e => generated 0 bytes in 427 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:20.770800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298b2dc7-eb54-478b-b042-40fe8425b8a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:20.771663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298b2dc7-eb54-478b-b042-40fe8425b8a9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:20.772150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298b2dc7-eb54-478b-b042-40fe8425b8a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:20.772556 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298b2dc7-eb54-478b-b042-40fe8425b8a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:20.773366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298b2dc7-eb54-478b-b042-40fe8425b8a9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:20.773800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298b2dc7-eb54-478b-b042-40fe8425b8a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:20.775339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298b2dc7-eb54-478b-b042-40fe8425b8a9 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:20.791155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 157/290] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:20.797146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831aa1cb-20a6-455b-9c4a-61b821a47233 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:20.921067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831aa1cb-20a6-455b-9c4a-61b821a47233 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:20.974880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2a257e-d57c-4096-a829-1c5785b76496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:20.975510 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2a257e-d57c-4096-a829-1c5785b76496 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:20.975782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2a257e-d57c-4096-a829-1c5785b76496 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:20.976045 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2a257e-d57c-4096-a829-1c5785b76496 None admin] PATH_INFO: `/v3/users/3c47c2755dfc48c38c37c96e25db3829` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:20.978874 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2a257e-d57c-4096-a829-1c5785b76496 None admin] RBAC: Authorizing `identity:get_user(user_id=3c47c2755dfc48c38c37c96e25db3829)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:20.980621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2a257e-d57c-4096-a829-1c5785b76496 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:20.982728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 134/291] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:20 2022] GET /identity/v3/users/3c47c2755dfc48c38c37c96e25db3829 => generated 297 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:20.989474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298b2dc7-eb54-478b-b042-40fe8425b8a9 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:21.015305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:21.117191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298b2dc7-eb54-478b-b042-40fe8425b8a9 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:21.161994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bba233-cd44-48db-94ae-3619456447f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:21.165064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bba233-cd44-48db-94ae-3619456447f9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:21.165336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bba233-cd44-48db-94ae-3619456447f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:21.165658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bba233-cd44-48db-94ae-3619456447f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:21.258183 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bba233-cd44-48db-94ae-3619456447f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:21.259694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bba233-cd44-48db-94ae-3619456447f9 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:21.292091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:21.384239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 158/292] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:21.395319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2a257e-d57c-4096-a829-1c5785b76496 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:21.475618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:21.496643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9302082-fbea-46f8-8a89-564d3812dfe1 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:21.496983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9302082-fbea-46f8-8a89-564d3812dfe1 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:21.497197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9302082-fbea-46f8-8a89-564d3812dfe1 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:21.498865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 159/293] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:21.504328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b6de56-a0df-4b26-aa58-15c8386229b3 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:21.504615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b6de56-a0df-4b26-aa58-15c8386229b3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:21.504892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b6de56-a0df-4b26-aa58-15c8386229b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:21.542956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-32b6de56-a0df-4b26-aa58-15c8386229b3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:21.544069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-32b6de56-a0df-4b26-aa58-15c8386229b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:21.583309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2a257e-d57c-4096-a829-1c5785b76496 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:21.618660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678aff91-dc2f-43bf-b2f8-536e7d18dea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:21.619287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678aff91-dc2f-43bf-b2f8-536e7d18dea4 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:21.620370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678aff91-dc2f-43bf-b2f8-536e7d18dea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:21.620681 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678aff91-dc2f-43bf-b2f8-536e7d18dea4 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:21.621798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678aff91-dc2f-43bf-b2f8-536e7d18dea4 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:21.623302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678aff91-dc2f-43bf-b2f8-536e7d18dea4 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:21.624922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 135/294] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:21 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:21.634059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678aff91-dc2f-43bf-b2f8-536e7d18dea4 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:21.668646 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 160/295] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:21.669986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e962ae38-24ff-4bd1-a8c0-e786d289fd5c None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:21.670306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e962ae38-24ff-4bd1-a8c0-e786d289fd5c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:21.670584 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e962ae38-24ff-4bd1-a8c0-e786d289fd5c None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:21.672039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 161/296] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:21.678430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa27c6b9-5c27-494a-85e0-7a9afd61e4c2 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:21.679144 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa27c6b9-5c27-494a-85e0-7a9afd61e4c2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:21.679431 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa27c6b9-5c27-494a-85e0-7a9afd61e4c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:21.702575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-aa27c6b9-5c27-494a-85e0-7a9afd61e4c2 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:21.704310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-aa27c6b9-5c27-494a-85e0-7a9afd61e4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:21.783633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:21.799214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678aff91-dc2f-43bf-b2f8-536e7d18dea4 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:21.825171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:21.842865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 162/297] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:21.854064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6eadf5-b3da-4fc4-9172-6efc539d9183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:21.854740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6eadf5-b3da-4fc4-9172-6efc539d9183 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:21.855017 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6eadf5-b3da-4fc4-9172-6efc539d9183 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:21.855294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6eadf5-b3da-4fc4-9172-6efc539d9183 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:21.856033 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6eadf5-b3da-4fc4-9172-6efc539d9183 None admin] RBAC: Adding query filter params (role.id=9912acfb698e41ebbce6a5147fafe896, scope.project.id=5c50437619304d0d857f65feed1d19e3, user.id=3c47c2755dfc48c38c37c96e25db3829) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:21.856341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6eadf5-b3da-4fc4-9172-6efc539d9183 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:21.857829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6eadf5-b3da-4fc4-9172-6efc539d9183 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:21.869146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 136/298] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:21 2022] GET /identity/v3/role_assignments?user.id=3c47c2755dfc48c38c37c96e25db3829&scope.project.id=5c50437619304d0d857f65feed1d19e3&role.id=9912acfb698e41ebbce6a5147fafe896 => generated 608 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:22.059203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa27c6b9-5c27-494a-85e0-7a9afd61e4c2 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:22.140688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6eadf5-b3da-4fc4-9172-6efc539d9183 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:22.174942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa27c6b9-5c27-494a-85e0-7a9afd61e4c2 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:22.220231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad5ae46-8a56-4cc2-b6c6-beb02c27a765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:22.221142 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad5ae46-8a56-4cc2-b6c6-beb02c27a765 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:22.221429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad5ae46-8a56-4cc2-b6c6-beb02c27a765 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:22.221703 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad5ae46-8a56-4cc2-b6c6-beb02c27a765 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:22.237219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad5ae46-8a56-4cc2-b6c6-beb02c27a765 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-6ad5ae46-8a56-4cc2-b6c6-beb02c27a765 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:22.245740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:22.247789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:22.249731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 163/299] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:22.256060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad5ae46-8a56-4cc2-b6c6-beb02c27a765 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:22.268535 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6eadf5-b3da-4fc4-9172-6efc539d9183 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:22.307305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:22.321928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692626e8-a47a-44e2-93d1-1c9f3aa0ac0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:22.321928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692626e8-a47a-44e2-93d1-1c9f3aa0ac0a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:22.321928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692626e8-a47a-44e2-93d1-1c9f3aa0ac0a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:22.321928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692626e8-a47a-44e2-93d1-1c9f3aa0ac0a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:22.322293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692626e8-a47a-44e2-93d1-1c9f3aa0ac0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:22.322293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692626e8-a47a-44e2-93d1-1c9f3aa0ac0a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:22.322293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 137/300] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:22.326448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692626e8-a47a-44e2-93d1-1c9f3aa0ac0a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:22.385570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:22.397051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad5ae46-8a56-4cc2-b6c6-beb02c27a765 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:22.438835 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc66277-f649-477e-85e0-5e570666cac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:22.439443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc66277-f649-477e-85e0-5e570666cac8 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:22.439725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc66277-f649-477e-85e0-5e570666cac8 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:22.439996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc66277-f649-477e-85e0-5e570666cac8 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:22.440665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc66277-f649-477e-85e0-5e570666cac8 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:22.441002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc66277-f649-477e-85e0-5e570666cac8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:22.442656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc66277-f649-477e-85e0-5e570666cac8 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:22.443399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692626e8-a47a-44e2-93d1-1c9f3aa0ac0a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:22.450876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 164/301] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:22.453679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ae51c0-d0ef-4858-a784-8a5ff338a97b None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:22.454654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ae51c0-d0ef-4858-a784-8a5ff338a97b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:22.455134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ae51c0-d0ef-4858-a784-8a5ff338a97b None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:22.457485 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 165/302] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:22 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:22.461700 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ae51c0-d0ef-4858-a784-8a5ff338a97b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:22.483637 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d60daa-cc15-46d9-abdf-aebbd4b6f51c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:22.484249 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d60daa-cc15-46d9-abdf-aebbd4b6f51c None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:22.484526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d60daa-cc15-46d9-abdf-aebbd4b6f51c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:22.484799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d60daa-cc15-46d9-abdf-aebbd4b6f51c None admin] PATH_INFO: `/v3/groups` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:22.485425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d60daa-cc15-46d9-abdf-aebbd4b6f51c None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:22.486895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d60daa-cc15-46d9-abdf-aebbd4b6f51c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:22.558934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:22.565445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ae51c0-d0ef-4858-a784-8a5ff338a97b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:22.583727 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 138/303] 10.222.0.65 () {60 vars in 1134 bytes} [Wed Aug 24 03:26:22 2022] POST /identity/v3/groups => generated 229 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:26:22.584905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a55e98-1bb9-40b4-8d27-dd61e19c94d3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:22.585188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a55e98-1bb9-40b4-8d27-dd61e19c94d3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:22.585457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a55e98-1bb9-40b4-8d27-dd61e19c94d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:22.610720 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-00a55e98-1bb9-40b4-8d27-dd61e19c94d3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:22.613638 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-00a55e98-1bb9-40b4-8d27-dd61e19c94d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:22.619032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae249f8-db2d-4e09-a535-d28cb6118d4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:22.619360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae249f8-db2d-4e09-a535-d28cb6118d4b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:22.619644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae249f8-db2d-4e09-a535-d28cb6118d4b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:22.619921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae249f8-db2d-4e09-a535-d28cb6118d4b None admin] PATH_INFO: `/v3/users/c35cae6e6f8848a9bf776b606a1b1f7f` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:22.637737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae249f8-db2d-4e09-a535-d28cb6118d4b None admin] RBAC: Authorizing `identity:get_user(user_id=c35cae6e6f8848a9bf776b606a1b1f7f)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:22.639856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae249f8-db2d-4e09-a535-d28cb6118d4b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:22.642137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 166/304] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:22 2022] GET /identity/v3/users/c35cae6e6f8848a9bf776b606a1b1f7f => generated 293 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:22.648371 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae249f8-db2d-4e09-a535-d28cb6118d4b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:22.711409 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 139/305] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:22.766787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae249f8-db2d-4e09-a535-d28cb6118d4b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:22.804048 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b57aa4f-b299-42df-ae66-b59e65991798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:22.804720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b57aa4f-b299-42df-ae66-b59e65991798 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:22.805042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b57aa4f-b299-42df-ae66-b59e65991798 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:22.805324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b57aa4f-b299-42df-ae66-b59e65991798 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:22.806057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b57aa4f-b299-42df-ae66-b59e65991798 None admin] RBAC: Adding query filter params (role.id=74e1fd5acabf47a7b788b2dc78d1fb3e, user.id=c35cae6e6f8848a9bf776b606a1b1f7f, scope.system=all) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:22.806437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b57aa4f-b299-42df-ae66-b59e65991798 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:22.807981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b57aa4f-b299-42df-ae66-b59e65991798 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:22.813694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 167/306] 10.222.0.65 () {56 vars in 1308 bytes} [Wed Aug 24 03:26:22 2022] GET /identity/v3/role_assignments?user.id=c35cae6e6f8848a9bf776b606a1b1f7f&scope.system=all&role.id=74e1fd5acabf47a7b788b2dc78d1fb3e => generated 226 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:22.986914 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a55e98-1bb9-40b4-8d27-dd61e19c94d3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:23.022835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:23.043422 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2dabdf-3c37-4d5a-8eb5-525fc3ccdbbd None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:23.043849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2dabdf-3c37-4d5a-8eb5-525fc3ccdbbd None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:23.044230 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2dabdf-3c37-4d5a-8eb5-525fc3ccdbbd None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:23.045971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 168/307] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:23.051678 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd9876d-5c30-4e5f-93e1-b7a4e5a913fc None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:23.052102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd9876d-5c30-4e5f-93e1-b7a4e5a913fc None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:23.052465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd9876d-5c30-4e5f-93e1-b7a4e5a913fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:23.071371 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-3bd9876d-5c30-4e5f-93e1-b7a4e5a913fc None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:23.072697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3bd9876d-5c30-4e5f-93e1-b7a4e5a913fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:23.090601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:23.093835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a55e98-1bb9-40b4-8d27-dd61e19c94d3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:23.133990 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fe4b11-aaf1-4a1a-b3d5-18bc66e23cbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:23.134676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fe4b11-aaf1-4a1a-b3d5-18bc66e23cbf None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:23.134955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fe4b11-aaf1-4a1a-b3d5-18bc66e23cbf None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:23.135230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fe4b11-aaf1-4a1a-b3d5-18bc66e23cbf None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:23.147384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fe4b11-aaf1-4a1a-b3d5-18bc66e23cbf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-a4fe4b11-aaf1-4a1a-b3d5-18bc66e23cbf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:23.152827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:23.153717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:23.154538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:23.154538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:23.154538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:23.154538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:23.154538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:23.154538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:23.154538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:23.154538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:23.154538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:23.154538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:23.154538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:23.154538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:23.158276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 140/308] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:23.158917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 169/309] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:26:23.163139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fe4b11-aaf1-4a1a-b3d5-18bc66e23cbf None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:23.265610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:23.274018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fe4b11-aaf1-4a1a-b3d5-18bc66e23cbf None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:23.308491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3faeb94a-5f1b-4b58-a955-e1c37495a7e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:23.309131 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faeb94a-5f1b-4b58-a955-e1c37495a7e7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:23.309416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faeb94a-5f1b-4b58-a955-e1c37495a7e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:23.309694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faeb94a-5f1b-4b58-a955-e1c37495a7e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:23.310425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3faeb94a-5f1b-4b58-a955-e1c37495a7e7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:23.310737 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3faeb94a-5f1b-4b58-a955-e1c37495a7e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:23.312116 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3faeb94a-5f1b-4b58-a955-e1c37495a7e7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:23.325360 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 141/310] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:23.326825 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd9876d-5c30-4e5f-93e1-b7a4e5a913fc None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:23.432929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd9876d-5c30-4e5f-93e1-b7a4e5a913fc None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:23.452294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3faeb94a-5f1b-4b58-a955-e1c37495a7e7 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:23.468231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faab8284-10e7-486a-b77a-1e0035e050f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:23.468875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faab8284-10e7-486a-b77a-1e0035e050f0 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:23.469154 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faab8284-10e7-486a-b77a-1e0035e050f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:23.469526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faab8284-10e7-486a-b77a-1e0035e050f0 None admin] PATH_INFO: `/v3/users/d2825a6523264021bf18ee593b51215e` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:23.470815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faab8284-10e7-486a-b77a-1e0035e050f0 None admin] RBAC: Authorizing `identity:get_user(user_id=d2825a6523264021bf18ee593b51215e)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:23.472350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faab8284-10e7-486a-b77a-1e0035e050f0 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:23.474146 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 170/311] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:23 2022] GET /identity/v3/users/d2825a6523264021bf18ee593b51215e => generated 293 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:23.480295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faab8284-10e7-486a-b77a-1e0035e050f0 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:23.502869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:23.590196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3faeb94a-5f1b-4b58-a955-e1c37495a7e7 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:23.610995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faab8284-10e7-486a-b77a-1e0035e050f0 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:23.632576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ba3b51-d290-456c-b76d-937a17471074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:23.633594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ba3b51-d290-456c-b76d-937a17471074 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:23.633876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ba3b51-d290-456c-b76d-937a17471074 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:23.634169 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ba3b51-d290-456c-b76d-937a17471074 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:23.648426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ba3b51-d290-456c-b76d-937a17471074 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:23.652348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627208ea-245a-4e69-b2ab-0727826102eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:23.653134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627208ea-245a-4e69-b2ab-0727826102eb None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:23.653616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627208ea-245a-4e69-b2ab-0727826102eb None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:23.653990 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627208ea-245a-4e69-b2ab-0727826102eb None admin] PATH_INFO: `/v3/system/users/d2825a6523264021bf18ee593b51215e/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-78ba3b51-d290-456c-b76d-937a17471074 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:23.654326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:23.655190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:23.655970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:23.655970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:23.655970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:23.655970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:23.655970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:23.655970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:23.655970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:23.655970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:23.655970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:23.655970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:23.655970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:23.655970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:23.656571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627208ea-245a-4e69-b2ab-0727826102eb None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=d2825a6523264021bf18ee593b51215e, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:23.656640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 142/312] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:23.657315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627208ea-245a-4e69-b2ab-0727826102eb None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:23.663399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ba3b51-d290-456c-b76d-937a17471074 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:23.683949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:23.751034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 171/313] 10.222.0.65 () {58 vars in 1334 bytes} [Wed Aug 24 03:26:23 2022] PUT /identity/v3/system/users/d2825a6523264021bf18ee593b51215e/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:26:23.775421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ba3b51-d290-456c-b76d-937a17471074 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:23.825499 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c71fe9-c9a1-4b38-8f38-8b8ba236af91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:23.826234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c71fe9-c9a1-4b38-8f38-8b8ba236af91 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:23.826527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c71fe9-c9a1-4b38-8f38-8b8ba236af91 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:23.826912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c71fe9-c9a1-4b38-8f38-8b8ba236af91 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:23.827663 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c71fe9-c9a1-4b38-8f38-8b8ba236af91 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:23.827979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c71fe9-c9a1-4b38-8f38-8b8ba236af91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:23.829408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c71fe9-c9a1-4b38-8f38-8b8ba236af91 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:23.840181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 143/314] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:23 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:23.845599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627208ea-245a-4e69-b2ab-0727826102eb None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:23.942851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:23.963439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627208ea-245a-4e69-b2ab-0727826102eb None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:23.998498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0ff470-b9e2-41d3-810c-0638402911d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:23.999124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0ff470-b9e2-41d3-810c-0638402911d0 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:23.999398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0ff470-b9e2-41d3-810c-0638402911d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:23.999667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0ff470-b9e2-41d3-810c-0638402911d0 None admin] PATH_INFO: `/v3/users/f59235a684414536a3d259252cb5b70d` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:24.000805 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0ff470-b9e2-41d3-810c-0638402911d0 None admin] RBAC: Authorizing `identity:get_user(user_id=f59235a684414536a3d259252cb5b70d)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:24.002306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0ff470-b9e2-41d3-810c-0638402911d0 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:24.004629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 172/315] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:23 2022] GET /identity/v3/users/f59235a684414536a3d259252cb5b70d => generated 297 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:24.010642 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c71fe9-c9a1-4b38-8f38-8b8ba236af91 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:24.097367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:24.117395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c71fe9-c9a1-4b38-8f38-8b8ba236af91 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:24.153457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2feb5a-d643-42ae-acbc-9acec21da9cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:24.154066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2feb5a-d643-42ae-acbc-9acec21da9cc None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:24.154414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2feb5a-d643-42ae-acbc-9acec21da9cc None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:24.154687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2feb5a-d643-42ae-acbc-9acec21da9cc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:24.225567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2feb5a-d643-42ae-acbc-9acec21da9cc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:24.227066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2feb5a-d643-42ae-acbc-9acec21da9cc None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:24.318697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:24.332186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 144/316] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:24.337791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0ff470-b9e2-41d3-810c-0638402911d0 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:24.449057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0ff470-b9e2-41d3-810c-0638402911d0 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:24.483957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78020776-b9ef-4114-abcc-3a5fee346fd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:24.484594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78020776-b9ef-4114-abcc-3a5fee346fd1 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:24.484870 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78020776-b9ef-4114-abcc-3a5fee346fd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:24.485134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78020776-b9ef-4114-abcc-3a5fee346fd1 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:24.486286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78020776-b9ef-4114-abcc-3a5fee346fd1 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:24.487725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78020776-b9ef-4114-abcc-3a5fee346fd1 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:24.490261 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 173/317] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:24 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:24.495904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2feb5a-d643-42ae-acbc-9acec21da9cc None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:24.599003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2feb5a-d643-42ae-acbc-9acec21da9cc None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:24.635876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f193994-216c-48e9-ab67-c672ee577e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:24.636478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f193994-216c-48e9-ab67-c672ee577e02 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:24.636756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f193994-216c-48e9-ab67-c672ee577e02 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:24.637024 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f193994-216c-48e9-ab67-c672ee577e02 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:24.637749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f193994-216c-48e9-ab67-c672ee577e02 None admin] RBAC: Adding query filter params (role.id=74e1fd5acabf47a7b788b2dc78d1fb3e, scope.project.id=5c50437619304d0d857f65feed1d19e3, user.id=f59235a684414536a3d259252cb5b70d) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:24.638052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f193994-216c-48e9-ab67-c672ee577e02 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:24.639520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f193994-216c-48e9-ab67-c672ee577e02 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:24.646286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 145/318] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:24 2022] GET /identity/v3/role_assignments?user.id=f59235a684414536a3d259252cb5b70d&scope.project.id=5c50437619304d0d857f65feed1d19e3&role.id=74e1fd5acabf47a7b788b2dc78d1fb3e => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:24.704817 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3d7f09-e5c6-4b30-8ab4-087d01f742a0 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:24.705442 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3d7f09-e5c6-4b30-8ab4-087d01f742a0 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:24.705906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3d7f09-e5c6-4b30-8ab4-087d01f742a0 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:24.708866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 174/319] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:24 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:24.714940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de352988-72ec-43dc-9573-a746b20dde36 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:24.715563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de352988-72ec-43dc-9573-a746b20dde36 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:24.716064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de352988-72ec-43dc-9573-a746b20dde36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:24.736101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-de352988-72ec-43dc-9573-a746b20dde36 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:24.737704 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-de352988-72ec-43dc-9573-a746b20dde36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:24.784926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:24.812223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 146/320] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:24.841644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f71928-6937-4983-a2d9-99ab729a1e2b None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:24.841996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f71928-6937-4983-a2d9-99ab729a1e2b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:24.842385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f71928-6937-4983-a2d9-99ab729a1e2b None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:24.844074 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 175/321] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:24.849661 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136c2120-6147-4668-ad4a-399e6202b1c9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:24.849952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136c2120-6147-4668-ad4a-399e6202b1c9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:24.850253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136c2120-6147-4668-ad4a-399e6202b1c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:24.870734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-136c2120-6147-4668-ad4a-399e6202b1c9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:24.872034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-136c2120-6147-4668-ad4a-399e6202b1c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:24.945342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 147/322] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:25.087448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f71928-6937-4983-a2d9-99ab729a1e2b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:25.107028 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:25.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f71928-6937-4983-a2d9-99ab729a1e2b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:25.216120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136c2120-6147-4668-ad4a-399e6202b1c9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:25.228633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a980c24-144d-421d-9b50-dee3130d62bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:25.229341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a980c24-144d-421d-9b50-dee3130d62bb None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:25.229627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a980c24-144d-421d-9b50-dee3130d62bb None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:25.229928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a980c24-144d-421d-9b50-dee3130d62bb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:25.231061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a980c24-144d-421d-9b50-dee3130d62bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:25.232511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a980c24-144d-421d-9b50-dee3130d62bb None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:25.234125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 176/323] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:25.240618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a980c24-144d-421d-9b50-dee3130d62bb None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:25.324576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136c2120-6147-4668-ad4a-399e6202b1c9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:25.327374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:25.333150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:25.350108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a980c24-144d-421d-9b50-dee3130d62bb None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:25.358855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d77cbd-125e-4260-aad6-c345b696fd12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:25.359485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d77cbd-125e-4260-aad6-c345b696fd12 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:25.359759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d77cbd-125e-4260-aad6-c345b696fd12 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:25.360019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d77cbd-125e-4260-aad6-c345b696fd12 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:25.371196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d77cbd-125e-4260-aad6-c345b696fd12 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-c4d77cbd-125e-4260-aad6-c345b696fd12 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:25.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:25.376936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:25.380006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 148/324] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:25.383567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d77cbd-125e-4260-aad6-c345b696fd12 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:25.387855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea09e0b2-73ae-4e9f-af85-4a02463e9c94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:25.388465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea09e0b2-73ae-4e9f-af85-4a02463e9c94 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:25.388771 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea09e0b2-73ae-4e9f-af85-4a02463e9c94 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:25.389144 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea09e0b2-73ae-4e9f-af85-4a02463e9c94 None admin] PATH_INFO: `/v3/groups` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:25.389909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea09e0b2-73ae-4e9f-af85-4a02463e9c94 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:25.391606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea09e0b2-73ae-4e9f-af85-4a02463e9c94 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:25.491169 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d77cbd-125e-4260-aad6-c345b696fd12 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:25.493947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 177/325] 10.222.0.65 () {60 vars in 1134 bytes} [Wed Aug 24 03:26:25 2022] POST /identity/v3/groups => generated 226 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:26:25.530078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11671982-8113-4c0d-98e3-ee9c245bf5c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:25.530762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11671982-8113-4c0d-98e3-ee9c245bf5c1 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:25.531029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11671982-8113-4c0d-98e3-ee9c245bf5c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:25.531300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11671982-8113-4c0d-98e3-ee9c245bf5c1 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:25.531977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11671982-8113-4c0d-98e3-ee9c245bf5c1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:25.532278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11671982-8113-4c0d-98e3-ee9c245bf5c1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:25.533705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11671982-8113-4c0d-98e3-ee9c245bf5c1 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:25.545308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 149/326] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:25.549339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea09e0b2-73ae-4e9f-af85-4a02463e9c94 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:25.661530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea09e0b2-73ae-4e9f-af85-4a02463e9c94 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:25.698113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6642208e-77ee-41d4-9dcc-0402b4965aca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:25.699132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6642208e-77ee-41d4-9dcc-0402b4965aca None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:25.699425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6642208e-77ee-41d4-9dcc-0402b4965aca None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:25.699675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6642208e-77ee-41d4-9dcc-0402b4965aca None admin] PATH_INFO: `/v3/users/c35cae6e6f8848a9bf776b606a1b1f7f` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:25.700815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6642208e-77ee-41d4-9dcc-0402b4965aca None admin] RBAC: Authorizing `identity:get_user(user_id=c35cae6e6f8848a9bf776b606a1b1f7f)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:25.702313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6642208e-77ee-41d4-9dcc-0402b4965aca None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:25.704169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 178/327] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:25 2022] GET /identity/v3/users/c35cae6e6f8848a9bf776b606a1b1f7f => generated 293 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:25.710879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11671982-8113-4c0d-98e3-ee9c245bf5c1 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:25.716521 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce72f2f-d7f6-4198-ac60-f0f419e1ec6f None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:25.716807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce72f2f-d7f6-4198-ac60-f0f419e1ec6f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:25.717082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce72f2f-d7f6-4198-ac60-f0f419e1ec6f None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:25.718489 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 179/328] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:26:25.723736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa8f6ed-844c-4f25-96b6-78692a02a0d2 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:25.724019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa8f6ed-844c-4f25-96b6-78692a02a0d2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:25.724293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa8f6ed-844c-4f25-96b6-78692a02a0d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:25.744929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:25.747377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1aa8f6ed-844c-4f25-96b6-78692a02a0d2 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:25.748593 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1aa8f6ed-844c-4f25-96b6-78692a02a0d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:25.756979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:25.832426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 180/329] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:26:25.839141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11671982-8113-4c0d-98e3-ee9c245bf5c1 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:25.876517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e863c4-2c30-482b-ae99-6dc617ca25c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:25.877173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e863c4-2c30-482b-ae99-6dc617ca25c4 None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:25.877451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e863c4-2c30-482b-ae99-6dc617ca25c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:25.877723 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e863c4-2c30-482b-ae99-6dc617ca25c4 None admin] PATH_INFO: `/v3/system/users/c35cae6e6f8848a9bf776b606a1b1f7f/roles/74e1fd5acabf47a7b788b2dc78d1fb3e` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:25.879821 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e863c4-2c30-482b-ae99-6dc617ca25c4 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=c35cae6e6f8848a9bf776b606a1b1f7f, role_id=74e1fd5acabf47a7b788b2dc78d1fb3e)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:25.881442 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e863c4-2c30-482b-ae99-6dc617ca25c4 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:25.934271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 150/330] 10.222.0.65 () {58 vars in 1334 bytes} [Wed Aug 24 03:26:25 2022] PUT /identity/v3/system/users/c35cae6e6f8848a9bf776b606a1b1f7f/roles/74e1fd5acabf47a7b788b2dc78d1fb3e => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:26.103465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa8f6ed-844c-4f25-96b6-78692a02a0d2 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:26.218443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa8f6ed-844c-4f25-96b6-78692a02a0d2 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:26.260959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efce0f98-056e-4ecb-8c94-37d27b347d5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:26.261615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efce0f98-056e-4ecb-8c94-37d27b347d5f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:26.261907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efce0f98-056e-4ecb-8c94-37d27b347d5f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:26.262220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efce0f98-056e-4ecb-8c94-37d27b347d5f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:26.275650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efce0f98-056e-4ecb-8c94-37d27b347d5f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-efce0f98-056e-4ecb-8c94-37d27b347d5f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:26.280548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:26.281401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:26.282251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 181/331] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:26.288773 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e863c4-2c30-482b-ae99-6dc617ca25c4 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:26.400122 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:26.410634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e863c4-2c30-482b-ae99-6dc617ca25c4 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:26.457170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15430d10-b67c-4415-8fe1-405d6f626dc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:26.457726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15430d10-b67c-4415-8fe1-405d6f626dc9 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:26.458022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15430d10-b67c-4415-8fe1-405d6f626dc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:26.458389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15430d10-b67c-4415-8fe1-405d6f626dc9 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:26.459058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15430d10-b67c-4415-8fe1-405d6f626dc9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:26.459359 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15430d10-b67c-4415-8fe1-405d6f626dc9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:26.460741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15430d10-b67c-4415-8fe1-405d6f626dc9 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:26.476333 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 151/332] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:26.481861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efce0f98-056e-4ecb-8c94-37d27b347d5f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:26.492469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:26.602407 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efce0f98-056e-4ecb-8c94-37d27b347d5f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:26.642866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4a165a-55f9-4fb0-a7fc-47a03666fe70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:26.643510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4a165a-55f9-4fb0-a7fc-47a03666fe70 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:26.643790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4a165a-55f9-4fb0-a7fc-47a03666fe70 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:26.644174 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4a165a-55f9-4fb0-a7fc-47a03666fe70 None admin] PATH_INFO: `/v3/users/d2825a6523264021bf18ee593b51215e` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:26.645964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4a165a-55f9-4fb0-a7fc-47a03666fe70 None admin] RBAC: Authorizing `identity:get_user(user_id=d2825a6523264021bf18ee593b51215e)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:26.647486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4a165a-55f9-4fb0-a7fc-47a03666fe70 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:26.649368 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 182/333] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:26 2022] GET /identity/v3/users/d2825a6523264021bf18ee593b51215e => generated 293 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:26.655490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15430d10-b67c-4415-8fe1-405d6f626dc9 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:26.786071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15430d10-b67c-4415-8fe1-405d6f626dc9 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:26.827718 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f1714d-f635-4993-91d4-3c58c51664c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:26.828336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f1714d-f635-4993-91d4-3c58c51664c6 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:26.828615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f1714d-f635-4993-91d4-3c58c51664c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:26.828887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f1714d-f635-4993-91d4-3c58c51664c6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:26.829634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f1714d-f635-4993-91d4-3c58c51664c6 None admin] RBAC: Adding query filter params (role.id=9912acfb698e41ebbce6a5147fafe896, user.id=d2825a6523264021bf18ee593b51215e, scope.system=all) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:26.829942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f1714d-f635-4993-91d4-3c58c51664c6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:26.831389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f1714d-f635-4993-91d4-3c58c51664c6 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:26.847984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 152/334] 10.222.0.65 () {56 vars in 1308 bytes} [Wed Aug 24 03:26:26 2022] GET /identity/v3/role_assignments?user.id=d2825a6523264021bf18ee593b51215e&scope.system=all&role.id=9912acfb698e41ebbce6a5147fafe896 => generated 510 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:28.130045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3f6b11-93c5-40c4-a77b-9e919a83f3fe None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:28.131336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3f6b11-93c5-40c4-a77b-9e919a83f3fe None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:28.131836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3f6b11-93c5-40c4-a77b-9e919a83f3fe None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:28.133852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 183/335] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:28 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:28.139160 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13a4cc1-2a6f-43e3-9422-b57e7d3ef450 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:28.139996 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13a4cc1-2a6f-43e3-9422-b57e7d3ef450 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:28.140468 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13a4cc1-2a6f-43e3-9422-b57e7d3ef450 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:28.164739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-a13a4cc1-2a6f-43e3-9422-b57e7d3ef450 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:28.166314 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a13a4cc1-2a6f-43e3-9422-b57e7d3ef450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:28.169884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:28.257478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 153/336] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:28.388935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856c1134-55c7-44e5-ba31-901a72ac68cf None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:28.389229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856c1134-55c7-44e5-ba31-901a72ac68cf None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:28.389409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856c1134-55c7-44e5-ba31-901a72ac68cf None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:28.391004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 184/337] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:28.397799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fac9ea5-715c-4c39-a257-032755b0b369 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:28.398086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fac9ea5-715c-4c39-a257-032755b0b369 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:28.398392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fac9ea5-715c-4c39-a257-032755b0b369 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:28.431112 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1fac9ea5-715c-4c39-a257-032755b0b369 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:28.432327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1fac9ea5-715c-4c39-a257-032755b0b369 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:28.524254 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 154/338] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:28.766070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856c1134-55c7-44e5-ba31-901a72ac68cf None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:28.792422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e619b9-e48a-4053-9b97-8a4bb3199e04 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:28.792752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e619b9-e48a-4053-9b97-8a4bb3199e04 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:28.792983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e619b9-e48a-4053-9b97-8a4bb3199e04 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:28.794518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 155/339] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:26:28.800938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d984898-54fa-4635-8f03-4fc28fbde5a2 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:28.801220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d984898-54fa-4635-8f03-4fc28fbde5a2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:28.801489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d984898-54fa-4635-8f03-4fc28fbde5a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:28.835488 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-4d984898-54fa-4635-8f03-4fc28fbde5a2 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:28.836766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4d984898-54fa-4635-8f03-4fc28fbde5a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:28.863455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:28.900908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856c1134-55c7-44e5-ba31-901a72ac68cf None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:28.939805 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8c14ca-302a-4a68-a643-0a5dda45362b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:28.940772 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8c14ca-302a-4a68-a643-0a5dda45362b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:28.941802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8c14ca-302a-4a68-a643-0a5dda45362b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:28.942310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8c14ca-302a-4a68-a643-0a5dda45362b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:28.954730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8c14ca-302a-4a68-a643-0a5dda45362b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:28.956731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-8b8c14ca-302a-4a68-a643-0a5dda45362b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:28.961253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:28.962141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:28.962953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:28.962953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:28.962953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:28.962953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:28.962953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:28.962953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:28.962953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:28.962953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:28.962953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:28.962953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:28.962953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:28.962953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:28.963549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 156/340] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:28.963549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d984898-54fa-4635-8f03-4fc28fbde5a2 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:28.965472 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 185/341] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:28.966932 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53f9bbe-fb52-4937-a13b-f6b2cbc62cb5 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:28.967366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53f9bbe-fb52-4937-a13b-f6b2cbc62cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:28.972783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53f9bbe-fb52-4937-a13b-f6b2cbc62cb5 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:28.972783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 186/342] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:28.974120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53f9bbe-fb52-4937-a13b-f6b2cbc62cb5 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:29.092176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d984898-54fa-4635-8f03-4fc28fbde5a2 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:29.095408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53f9bbe-fb52-4937-a13b-f6b2cbc62cb5 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:29.134624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42299e61-f930-4b16-a157-23ded37f2e94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:29.135260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42299e61-f930-4b16-a157-23ded37f2e94 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:29.135531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42299e61-f930-4b16-a157-23ded37f2e94 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:29.135795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42299e61-f930-4b16-a157-23ded37f2e94 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:29.136091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:29.144228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3198f61-e3d1-4f97-9c02-c47949855944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:29.145283 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3198f61-e3d1-4f97-9c02-c47949855944 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:29.146039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3198f61-e3d1-4f97-9c02-c47949855944 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:29.146526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3198f61-e3d1-4f97-9c02-c47949855944 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:29.147727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3198f61-e3d1-4f97-9c02-c47949855944 None admin] RBAC: Adding query filter params (name=member) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:29.157471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3198f61-e3d1-4f97-9c02-c47949855944 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:29.157600 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42299e61-f930-4b16-a157-23ded37f2e94 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:29.157998 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3198f61-e3d1-4f97-9c02-c47949855944 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-42299e61-f930-4b16-a157-23ded37f2e94 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:29.158069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:29.158910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 157/343] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:26:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:29.159776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d68febb-44eb-4506-adc4-7d57e269f474 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:29.160585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d68febb-44eb-4506-adc4-7d57e269f474 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:29.160585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d68febb-44eb-4506-adc4-7d57e269f474 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:29.162024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 187/344] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:29.167734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3198f61-e3d1-4f97-9c02-c47949855944 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:29.179486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-8d68febb-44eb-4506-adc4-7d57e269f474 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:29.180991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-8d68febb-44eb-4506-adc4-7d57e269f474 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:29.249707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:29.267056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 158/345] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:29.271282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d68febb-44eb-4506-adc4-7d57e269f474 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:29.302034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3198f61-e3d1-4f97-9c02-c47949855944 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:29.343936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191764af-3273-449b-a7e1-5e1f7f976df0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:29.344906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191764af-3273-449b-a7e1-5e1f7f976df0 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:29.345453 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191764af-3273-449b-a7e1-5e1f7f976df0 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:29.345866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191764af-3273-449b-a7e1-5e1f7f976df0 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:29.346724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191764af-3273-449b-a7e1-5e1f7f976df0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:29.347630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191764af-3273-449b-a7e1-5e1f7f976df0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:29.349752 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191764af-3273-449b-a7e1-5e1f7f976df0 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:29.359776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 188/346] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:29 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:29.365421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191764af-3273-449b-a7e1-5e1f7f976df0 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:29.473137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:29.484086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d68febb-44eb-4506-adc4-7d57e269f474 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:29.530958 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191764af-3273-449b-a7e1-5e1f7f976df0 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:29.555403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577d77c2-e597-406d-af45-f74d4eb55a9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:29.556022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577d77c2-e597-406d-af45-f74d4eb55a9e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:29.556296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577d77c2-e597-406d-af45-f74d4eb55a9e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:29.556563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577d77c2-e597-406d-af45-f74d4eb55a9e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:29.572109 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2212e7cb-57e6-4e25-a5d8-d818a52fec96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:29.573058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2212e7cb-57e6-4e25-a5d8-d818a52fec96 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:29.573681 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2212e7cb-57e6-4e25-a5d8-d818a52fec96 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:29.574144 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2212e7cb-57e6-4e25-a5d8-d818a52fec96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:29.651825 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:29.653975 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2212e7cb-57e6-4e25-a5d8-d818a52fec96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:29.655477 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2212e7cb-57e6-4e25-a5d8-d818a52fec96 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:29.674141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577d77c2-e597-406d-af45-f74d4eb55a9e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:29.676629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577d77c2-e597-406d-af45-f74d4eb55a9e None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:29.718254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:29.775510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 189/347] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 03:26:29.776202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2212e7cb-57e6-4e25-a5d8-d818a52fec96 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:29.812546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 159/348] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 546 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:29.814319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b0955f-a241-4d3d-a80f-9d155e13b753 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:29.814319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b0955f-a241-4d3d-a80f-9d155e13b753 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:29.814486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b0955f-a241-4d3d-a80f-9d155e13b753 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:29.815731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 160/349] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:29.816827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b0955f-a241-4d3d-a80f-9d155e13b753 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:29.910644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2212e7cb-57e6-4e25-a5d8-d818a52fec96 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:29.927597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:29.945831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b0955f-a241-4d3d-a80f-9d155e13b753 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:29.961808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de91a8f2-7980-4c4c-a0f2-9087e3cd17d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:29.963120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de91a8f2-7980-4c4c-a0f2-9087e3cd17d3 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:29.963819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de91a8f2-7980-4c4c-a0f2-9087e3cd17d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:29.964290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de91a8f2-7980-4c4c-a0f2-9087e3cd17d3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:29.977238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de91a8f2-7980-4c4c-a0f2-9087e3cd17d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-de91a8f2-7980-4c4c-a0f2-9087e3cd17d3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:29.982042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:29.982966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:29.983776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 190/350] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:29.984423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de91a8f2-7980-4c4c-a0f2-9087e3cd17d3 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:29.993180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1640e2ee-f8e7-4e9c-8388-5f673746d09f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:29.993819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1640e2ee-f8e7-4e9c-8388-5f673746d09f None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:29.994092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1640e2ee-f8e7-4e9c-8388-5f673746d09f None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:29.994469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1640e2ee-f8e7-4e9c-8388-5f673746d09f None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:29.995954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1640e2ee-f8e7-4e9c-8388-5f673746d09f None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:29.997756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1640e2ee-f8e7-4e9c-8388-5f673746d09f None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:30.000358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 161/351] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:29 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:30.001363 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1640e2ee-f8e7-4e9c-8388-5f673746d09f None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:30.006514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:30.100725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de91a8f2-7980-4c4c-a0f2-9087e3cd17d3 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:30.140812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1640e2ee-f8e7-4e9c-8388-5f673746d09f None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:30.146312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd55dee-a8ac-4fa1-9216-80d11f31f5ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:30.146889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd55dee-a8ac-4fa1-9216-80d11f31f5ec None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:30.148040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd55dee-a8ac-4fa1-9216-80d11f31f5ec None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:30.148040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd55dee-a8ac-4fa1-9216-80d11f31f5ec None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:30.162753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd55dee-a8ac-4fa1-9216-80d11f31f5ec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-acd55dee-a8ac-4fa1-9216-80d11f31f5ec None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:30.169673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:30.170755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:30.171746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:30.171746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:30.171746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:30.171746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:30.171746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:30.171746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:30.171746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:30.171746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:30.171746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:30.171746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:30.171746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 24 03:26:30.171746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:30.174027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 191/352] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:29 2022] GET /identity/v3/roles/reader => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:30.179577 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6120f5-2245-4c30-a596-8ff1f2b97fda None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:30.180034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6120f5-2245-4c30-a596-8ff1f2b97fda None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:30.180442 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6120f5-2245-4c30-a596-8ff1f2b97fda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:30.182334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a9f7b4-239c-440f-8d93-c54d4727e72b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:30.183048 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a9f7b4-239c-440f-8d93-c54d4727e72b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:30.183396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a9f7b4-239c-440f-8d93-c54d4727e72b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:30.183672 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a9f7b4-239c-440f-8d93-c54d4727e72b None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:30.185931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a9f7b4-239c-440f-8d93-c54d4727e72b None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:30.187401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a9f7b4-239c-440f-8d93-c54d4727e72b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:30.189065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 162/353] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:30 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:30.189850 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a9f7b4-239c-440f-8d93-c54d4727e72b None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:30.201029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-7f6120f5-2245-4c30-a596-8ff1f2b97fda None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:30.202477 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7f6120f5-2245-4c30-a596-8ff1f2b97fda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:30.224679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:30.292507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 192/354] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:30.293304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6120f5-2245-4c30-a596-8ff1f2b97fda None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:30.296609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:30.319146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a9f7b4-239c-440f-8d93-c54d4727e72b None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:30.356908 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fcab6b-f872-42c3-8b98-e3ec2a9d1634 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:30.357870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fcab6b-f872-42c3-8b98-e3ec2a9d1634 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:30.358392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fcab6b-f872-42c3-8b98-e3ec2a9d1634 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:30.358853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fcab6b-f872-42c3-8b98-e3ec2a9d1634 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:30.359694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fcab6b-f872-42c3-8b98-e3ec2a9d1634 None admin] RBAC: Adding query filter params (name=member) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:30.360179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fcab6b-f872-42c3-8b98-e3ec2a9d1634 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:30.361864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fcab6b-f872-42c3-8b98-e3ec2a9d1634 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:30.371347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 163/355] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:30.372545 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fcab6b-f872-42c3-8b98-e3ec2a9d1634 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:30.402905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6120f5-2245-4c30-a596-8ff1f2b97fda None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:30.442364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c041ad-1e6f-47cd-871f-c1fdeae056bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:30.443010 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c041ad-1e6f-47cd-871f-c1fdeae056bb None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:30.443291 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c041ad-1e6f-47cd-871f-c1fdeae056bb None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:30.443562 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c041ad-1e6f-47cd-871f-c1fdeae056bb None admin] PATH_INFO: `/v3/groups/f95b8af0f9ae46c7b420273e832ffb67` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:30.450562 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c041ad-1e6f-47cd-871f-c1fdeae056bb None admin] RBAC: Authorizing `identity:get_group(group_id=f95b8af0f9ae46c7b420273e832ffb67)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:30.452037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c041ad-1e6f-47cd-871f-c1fdeae056bb None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:30.453987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 193/356] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:30 2022] GET /identity/v3/groups/f95b8af0f9ae46c7b420273e832ffb67 => generated 226 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:30.454869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c041ad-1e6f-47cd-871f-c1fdeae056bb None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:30.471524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:30.485105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fcab6b-f872-42c3-8b98-e3ec2a9d1634 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:30.523555 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f9b90e-af8b-4959-810e-9a62fa10488a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:30.524449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f9b90e-af8b-4959-810e-9a62fa10488a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:30.524912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f9b90e-af8b-4959-810e-9a62fa10488a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:30.525315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f9b90e-af8b-4959-810e-9a62fa10488a None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:30.526169 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f9b90e-af8b-4959-810e-9a62fa10488a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:30.526634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f9b90e-af8b-4959-810e-9a62fa10488a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:30.528139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f9b90e-af8b-4959-810e-9a62fa10488a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:30.528679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:30.538553 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 164/357] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:30 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:30.539700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f9b90e-af8b-4959-810e-9a62fa10488a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:30.570489 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c041ad-1e6f-47cd-871f-c1fdeae056bb None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:30.609213 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad777565-77d3-44b6-a3bb-4e96ee170af4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:30.609852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad777565-77d3-44b6-a3bb-4e96ee170af4 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:30.610480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad777565-77d3-44b6-a3bb-4e96ee170af4 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:30.610480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad777565-77d3-44b6-a3bb-4e96ee170af4 None admin] PATH_INFO: `/v3/groups/f95b8af0f9ae46c7b420273e832ffb67` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:30.611558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad777565-77d3-44b6-a3bb-4e96ee170af4 None admin] RBAC: Authorizing `identity:get_group(group_id=f95b8af0f9ae46c7b420273e832ffb67)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:30.613561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad777565-77d3-44b6-a3bb-4e96ee170af4 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:30.615187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 194/358] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:30 2022] GET /identity/v3/groups/f95b8af0f9ae46c7b420273e832ffb67 => generated 226 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:30.616094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad777565-77d3-44b6-a3bb-4e96ee170af4 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:30.656208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f9b90e-af8b-4959-810e-9a62fa10488a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:30.693336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3926d5-cb2e-4868-b8af-f2247bf5858b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:30.693869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3926d5-cb2e-4868-b8af-f2247bf5858b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:30.694143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3926d5-cb2e-4868-b8af-f2247bf5858b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:30.694441 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3926d5-cb2e-4868-b8af-f2247bf5858b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:30.716498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:30.730643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad777565-77d3-44b6-a3bb-4e96ee170af4 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:30.770370 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9467cd54-9bf6-4ed8-b48c-f693b0deb6b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:30.770982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9467cd54-9bf6-4ed8-b48c-f693b0deb6b6 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:30.771259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9467cd54-9bf6-4ed8-b48c-f693b0deb6b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:30.771531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9467cd54-9bf6-4ed8-b48c-f693b0deb6b6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:30.772262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9467cd54-9bf6-4ed8-b48c-f693b0deb6b6 None admin] RBAC: Adding query filter params (group.id=f95b8af0f9ae46c7b420273e832ffb67, role.id=d02a8e6c49b74aaa9403f8bd535b90bf, scope.project.id=db44137c64a347cfb350dad24d36a3db) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:30.772568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9467cd54-9bf6-4ed8-b48c-f693b0deb6b6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:30.773142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3926d5-cb2e-4868-b8af-f2247bf5858b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:30.774137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9467cd54-9bf6-4ed8-b48c-f693b0deb6b6 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:30.774617 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3926d5-cb2e-4868-b8af-f2247bf5858b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:30.778543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:30.780240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 195/359] 10.222.0.65 () {56 vars in 1376 bytes} [Wed Aug 24 03:26:30 2022] GET /identity/v3/role_assignments?group.id=f95b8af0f9ae46c7b420273e832ffb67&scope.project.id=db44137c64a347cfb350dad24d36a3db&role.id=d02a8e6c49b74aaa9403f8bd535b90bf => generated 260 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:30.782094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9467cd54-9bf6-4ed8-b48c-f693b0deb6b6 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:30.911193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 165/360] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:30.912877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bc935-f527-4c64-b3ac-6c4bd72b7b4b None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:30.913346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bc935-f527-4c64-b3ac-6c4bd72b7b4b None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:30.913815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bc935-f527-4c64-b3ac-6c4bd72b7b4b None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:30.915652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 166/361] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:30.916594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268bc935-f527-4c64-b3ac-6c4bd72b7b4b None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:30.934552 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9467cd54-9bf6-4ed8-b48c-f693b0deb6b6 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:30.983253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:30.984708 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca740a93-419d-4235-8b4e-549022522e8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:30.985355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca740a93-419d-4235-8b4e-549022522e8d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:30.985636 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca740a93-419d-4235-8b4e-549022522e8d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:30.986407 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca740a93-419d-4235-8b4e-549022522e8d None admin] PATH_INFO: `/v3/users/c35cae6e6f8848a9bf776b606a1b1f7f` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:30.987583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca740a93-419d-4235-8b4e-549022522e8d None admin] RBAC: Authorizing `identity:get_user(user_id=c35cae6e6f8848a9bf776b606a1b1f7f)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:30.989578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca740a93-419d-4235-8b4e-549022522e8d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:30.991196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 196/362] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:26:30 2022] GET /identity/v3/users/c35cae6e6f8848a9bf776b606a1b1f7f => generated 293 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:30.991962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca740a93-419d-4235-8b4e-549022522e8d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:31.053008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268bc935-f527-4c64-b3ac-6c4bd72b7b4b None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:31.087266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:31.098898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8c1c61-23b1-450c-a4fc-a60c50c7d69c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:31.099861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8c1c61-23b1-450c-a4fc-a60c50c7d69c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:31.100276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8c1c61-23b1-450c-a4fc-a60c50c7d69c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:31.100678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8c1c61-23b1-450c-a4fc-a60c50c7d69c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:31.125070 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8c1c61-23b1-450c-a4fc-a60c50c7d69c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-8f8c1c61-23b1-450c-a4fc-a60c50c7d69c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:31.131998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:31.132901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:31.133688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:31.133688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:31.133688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:31.133688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:31.133688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:31.133688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:31.133688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:31.133688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:31.133688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:31.133688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:31.133688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:31.133688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:31.134804 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca740a93-419d-4235-8b4e-549022522e8d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:31.135598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 167/363] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:26:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:31.136893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40092718-7cf4-4119-a3d0-cbd0fc55143c None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:31.138073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40092718-7cf4-4119-a3d0-cbd0fc55143c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:31.138378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40092718-7cf4-4119-a3d0-cbd0fc55143c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:31.162935 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-40092718-7cf4-4119-a3d0-cbd0fc55143c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:31.165130 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-40092718-7cf4-4119-a3d0-cbd0fc55143c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:31.178002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b06016-9d74-4e84-9755-5d9894534461 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:31.178754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b06016-9d74-4e84-9755-5d9894534461 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:31.179030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b06016-9d74-4e84-9755-5d9894534461 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:31.179298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b06016-9d74-4e84-9755-5d9894534461 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:31.180030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b06016-9d74-4e84-9755-5d9894534461 None admin] RBAC: Adding query filter params (group.id=f95b8af0f9ae46c7b420273e832ffb67, role.id=9912acfb698e41ebbce6a5147fafe896, scope.project.id=db44137c64a347cfb350dad24d36a3db) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:31.180334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b06016-9d74-4e84-9755-5d9894534461 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:31.181963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b06016-9d74-4e84-9755-5d9894534461 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:31.198303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 197/364] 10.222.0.65 () {56 vars in 1376 bytes} [Wed Aug 24 03:26:30 2022] GET /identity/v3/role_assignments?group.id=f95b8af0f9ae46c7b420273e832ffb67&scope.project.id=db44137c64a347cfb350dad24d36a3db&role.id=9912acfb698e41ebbce6a5147fafe896 => generated 260 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:31.199342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b06016-9d74-4e84-9755-5d9894534461 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:31.255249 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 168/365] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:31.256292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40092718-7cf4-4119-a3d0-cbd0fc55143c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:31.322863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b06016-9d74-4e84-9755-5d9894534461 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:31.328473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:31.356948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:31.362323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c3f91c-b53f-40b4-905c-4f8a62f68561 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:31.363807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3f91c-b53f-40b4-905c-4f8a62f68561 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:31.364329 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3f91c-b53f-40b4-905c-4f8a62f68561 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:31.364792 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3f91c-b53f-40b4-905c-4f8a62f68561 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:31.366169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c3f91c-b53f-40b4-905c-4f8a62f68561 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:31.367811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c3f91c-b53f-40b4-905c-4f8a62f68561 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:31.369951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 198/366] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:31 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:31.370952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c3f91c-b53f-40b4-905c-4f8a62f68561 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:31.427295 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40092718-7cf4-4119-a3d0-cbd0fc55143c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:31.481586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d44f4b-6d53-43d3-931f-5937628c5fb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:31.482229 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d44f4b-6d53-43d3-931f-5937628c5fb7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:31.482362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c3f91c-b53f-40b4-905c-4f8a62f68561 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:31.482541 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d44f4b-6d53-43d3-931f-5937628c5fb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:31.482775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d44f4b-6d53-43d3-931f-5937628c5fb7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:31.483516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d44f4b-6d53-43d3-931f-5937628c5fb7 None admin] RBAC: Adding query filter params (role.id=74e1fd5acabf47a7b788b2dc78d1fb3e, user.id=c35cae6e6f8848a9bf776b606a1b1f7f, scope.system=all) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:31.483822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d44f4b-6d53-43d3-931f-5937628c5fb7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:31.485331 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d44f4b-6d53-43d3-931f-5937628c5fb7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:31.507312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 169/367] 10.222.0.65 () {56 vars in 1308 bytes} [Wed Aug 24 03:26:31 2022] GET /identity/v3/role_assignments?user.id=c35cae6e6f8848a9bf776b606a1b1f7f&scope.system=all&role.id=74e1fd5acabf47a7b788b2dc78d1fb3e => generated 510 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:31.508104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d44f4b-6d53-43d3-931f-5937628c5fb7 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:31.520100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233286e4-b2c0-4c25-a67a-48ab79725602 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:31.520896 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233286e4-b2c0-4c25-a67a-48ab79725602 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:31.521175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233286e4-b2c0-4c25-a67a-48ab79725602 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:31.521443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233286e4-b2c0-4c25-a67a-48ab79725602 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:31.522427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233286e4-b2c0-4c25-a67a-48ab79725602 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:31.522962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233286e4-b2c0-4c25-a67a-48ab79725602 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:31.525445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233286e4-b2c0-4c25-a67a-48ab79725602 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:31.536188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 199/368] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:31.543980 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233286e4-b2c0-4c25-a67a-48ab79725602 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:31.618085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:31.664608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233286e4-b2c0-4c25-a67a-48ab79725602 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:31.674222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:31.679523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d44f4b-6d53-43d3-931f-5937628c5fb7 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:31.720009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb583888-1c6a-4c8a-b9b0-ff2ac7c7cd00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:31.720699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb583888-1c6a-4c8a-b9b0-ff2ac7c7cd00 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:31.720979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb583888-1c6a-4c8a-b9b0-ff2ac7c7cd00 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:31.721355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb583888-1c6a-4c8a-b9b0-ff2ac7c7cd00 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:31.722978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e17c56-bc37-4732-93cc-e05b378d1e69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:31.723630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e17c56-bc37-4732-93cc-e05b378d1e69 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:31.723912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e17c56-bc37-4732-93cc-e05b378d1e69 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:31.724189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e17c56-bc37-4732-93cc-e05b378d1e69 None admin] PATH_INFO: `/v3/groups/f95b8af0f9ae46c7b420273e832ffb67` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:31.725412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e17c56-bc37-4732-93cc-e05b378d1e69 None admin] RBAC: Authorizing `identity:get_group(group_id=f95b8af0f9ae46c7b420273e832ffb67)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:31.726904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e17c56-bc37-4732-93cc-e05b378d1e69 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:31.728529 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 170/369] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:31 2022] GET /identity/v3/groups/f95b8af0f9ae46c7b420273e832ffb67 => generated 226 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:31.734912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e17c56-bc37-4732-93cc-e05b378d1e69 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:31.822791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb583888-1c6a-4c8a-b9b0-ff2ac7c7cd00 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:31.824214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb583888-1c6a-4c8a-b9b0-ff2ac7c7cd00 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:31.844753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e17c56-bc37-4732-93cc-e05b378d1e69 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:31.883176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9864e78-1270-42d9-a26a-c345ab6a74c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:31.883793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9864e78-1270-42d9-a26a-c345ab6a74c5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:31.884073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9864e78-1270-42d9-a26a-c345ab6a74c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:31.884348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9864e78-1270-42d9-a26a-c345ab6a74c5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:31.885084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9864e78-1270-42d9-a26a-c345ab6a74c5 None admin] RBAC: Adding query filter params (group.id=f95b8af0f9ae46c7b420273e832ffb67, role.id=9912acfb698e41ebbce6a5147fafe896, scope.project.id=5c50437619304d0d857f65feed1d19e3) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:31.885392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9864e78-1270-42d9-a26a-c345ab6a74c5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:31.886874 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9864e78-1270-42d9-a26a-c345ab6a74c5 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:31.894528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:31.894907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 171/370] 10.222.0.65 () {56 vars in 1376 bytes} [Wed Aug 24 03:26:31 2022] GET /identity/v3/role_assignments?group.id=f95b8af0f9ae46c7b420273e832ffb67&scope.project.id=5c50437619304d0d857f65feed1d19e3&role.id=9912acfb698e41ebbce6a5147fafe896 => generated 260 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:31.945780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 200/371] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:31.951785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9864e78-1270-42d9-a26a-c345ab6a74c5 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:31.979855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb583888-1c6a-4c8a-b9b0-ff2ac7c7cd00 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:32.002059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:32.155378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9864e78-1270-42d9-a26a-c345ab6a74c5 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:32.183825 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb583888-1c6a-4c8a-b9b0-ff2ac7c7cd00 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:32.225368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c91d20-35d5-44ed-abd6-e27c2cd21492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:32.226549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c91d20-35d5-44ed-abd6-e27c2cd21492 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:32.227027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c91d20-35d5-44ed-abd6-e27c2cd21492 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:32.227492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c91d20-35d5-44ed-abd6-e27c2cd21492 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:32.230764 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb884cd2-23aa-46ce-ae4f-191d77e5825f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:32.231114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c91d20-35d5-44ed-abd6-e27c2cd21492 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:32.231423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb884cd2-23aa-46ce-ae4f-191d77e5825f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:32.231673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb884cd2-23aa-46ce-ae4f-191d77e5825f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:32.231942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb884cd2-23aa-46ce-ae4f-191d77e5825f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:32.233434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c91d20-35d5-44ed-abd6-e27c2cd21492 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:32.235324 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 172/372] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:31 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:32.247697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb884cd2-23aa-46ce-ae4f-191d77e5825f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-cb884cd2-23aa-46ce-ae4f-191d77e5825f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:32.252699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:32.253651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:32.255047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 201/373] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:26:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:32.260544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c91d20-35d5-44ed-abd6-e27c2cd21492 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:32.260973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb884cd2-23aa-46ce-ae4f-191d77e5825f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:32.289033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:32.370219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb884cd2-23aa-46ce-ae4f-191d77e5825f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:32.406042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008d921e-84ff-4c29-9133-edb3891bcbbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:32.406599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008d921e-84ff-4c29-9133-edb3891bcbbc None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:32.406874 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008d921e-84ff-4c29-9133-edb3891bcbbc None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:32.407144 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008d921e-84ff-4c29-9133-edb3891bcbbc None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:32.410103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008d921e-84ff-4c29-9133-edb3891bcbbc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:32.410103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008d921e-84ff-4c29-9133-edb3891bcbbc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:32.410103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008d921e-84ff-4c29-9133-edb3891bcbbc None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:32.418877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 202/374] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:32.424200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008d921e-84ff-4c29-9133-edb3891bcbbc None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:32.503603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:32.508498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c91d20-35d5-44ed-abd6-e27c2cd21492 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:32.524131 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:32.538018 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008d921e-84ff-4c29-9133-edb3891bcbbc None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:32.562884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c0b594-4407-446d-9c1d-3689922305d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:32.563536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c0b594-4407-446d-9c1d-3689922305d3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:32.563823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c0b594-4407-446d-9c1d-3689922305d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:32.564102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c0b594-4407-446d-9c1d-3689922305d3 None admin] PATH_INFO: `/v3/groups/f95b8af0f9ae46c7b420273e832ffb67` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:32.565276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c0b594-4407-446d-9c1d-3689922305d3 None admin] RBAC: Authorizing `identity:get_group(group_id=f95b8af0f9ae46c7b420273e832ffb67)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:32.567484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c0b594-4407-446d-9c1d-3689922305d3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:32.570748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 173/375] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:32 2022] GET /identity/v3/groups/f95b8af0f9ae46c7b420273e832ffb67 => generated 226 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:32.576453 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c0b594-4407-446d-9c1d-3689922305d3 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:32.578779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264ce479-de03-4761-be74-4f5f7beb1e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:32.579500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264ce479-de03-4761-be74-4f5f7beb1e8f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:32.579784 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264ce479-de03-4761-be74-4f5f7beb1e8f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:32.580055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264ce479-de03-4761-be74-4f5f7beb1e8f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:32.663617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264ce479-de03-4761-be74-4f5f7beb1e8f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:32.665011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264ce479-de03-4761-be74-4f5f7beb1e8f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:32.694080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c0b594-4407-446d-9c1d-3689922305d3 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:32.732070 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc57dbe-3b06-4b6e-9a91-a7bee8dbe614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:32.732784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc57dbe-3b06-4b6e-9a91-a7bee8dbe614 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:32.733071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc57dbe-3b06-4b6e-9a91-a7bee8dbe614 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:32.733351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc57dbe-3b06-4b6e-9a91-a7bee8dbe614 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:32.735111 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc57dbe-3b06-4b6e-9a91-a7bee8dbe614 None admin] RBAC: Adding query filter params (group.id=f95b8af0f9ae46c7b420273e832ffb67, role.id=d02a8e6c49b74aaa9403f8bd535b90bf, scope.project.id=5c50437619304d0d857f65feed1d19e3) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:32.735446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc57dbe-3b06-4b6e-9a91-a7bee8dbe614 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:32.736997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc57dbe-3b06-4b6e-9a91-a7bee8dbe614 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:32.744565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 174/376] 10.222.0.65 () {56 vars in 1376 bytes} [Wed Aug 24 03:26:32 2022] GET /identity/v3/role_assignments?group.id=f95b8af0f9ae46c7b420273e832ffb67&scope.project.id=5c50437619304d0d857f65feed1d19e3&role.id=d02a8e6c49b74aaa9403f8bd535b90bf => generated 260 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:32.788594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:32.806202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 203/377] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 03:26:32.815330 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc57dbe-3b06-4b6e-9a91-a7bee8dbe614 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:32.865113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:32.953140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc57dbe-3b06-4b6e-9a91-a7bee8dbe614 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:32.992054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8465f7f-04c0-4a14-a17c-34e517dfe19a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:32.992690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8465f7f-04c0-4a14-a17c-34e517dfe19a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:32.992967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8465f7f-04c0-4a14-a17c-34e517dfe19a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:32.993241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8465f7f-04c0-4a14-a17c-34e517dfe19a None admin] PATH_INFO: `/v3/projects/b8d28080f87c4783b9ef971b31a0d51a` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:32.994414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8465f7f-04c0-4a14-a17c-34e517dfe19a None admin] RBAC: Authorizing `identity:get_project(project_id=b8d28080f87c4783b9ef971b31a0d51a)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:32.995847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8465f7f-04c0-4a14-a17c-34e517dfe19a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:32.997809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 175/378] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:32 2022] GET /identity/v3/projects/b8d28080f87c4783b9ef971b31a0d51a => generated 344 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:33.003061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264ce479-de03-4761-be74-4f5f7beb1e8f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:33.134303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264ce479-de03-4761-be74-4f5f7beb1e8f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:33.176277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee77f465-1b5c-470f-bdc3-16839d79a994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:33.176925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee77f465-1b5c-470f-bdc3-16839d79a994 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:33.177203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee77f465-1b5c-470f-bdc3-16839d79a994 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:33.177543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee77f465-1b5c-470f-bdc3-16839d79a994 None admin] PATH_INFO: `/v3/groups/2750e7b66ccf40bbb43b081af3edde7b` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:33.185668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee77f465-1b5c-470f-bdc3-16839d79a994 None admin] RBAC: Authorizing `identity:get_group(group_id=2750e7b66ccf40bbb43b081af3edde7b)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:33.187153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee77f465-1b5c-470f-bdc3-16839d79a994 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:33.189107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 204/379] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:33 2022] GET /identity/v3/groups/2750e7b66ccf40bbb43b081af3edde7b => generated 229 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:33.195131 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8465f7f-04c0-4a14-a17c-34e517dfe19a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:33.258528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523e6dc1-55bf-4b80-856d-6f77a0192b9b None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:33.258817 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523e6dc1-55bf-4b80-856d-6f77a0192b9b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:33.259003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523e6dc1-55bf-4b80-856d-6f77a0192b9b None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:33.260619 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 205/380] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:33.265907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76a9f3c-5f45-4dd1-9046-01ca857be052 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:33.266221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76a9f3c-5f45-4dd1-9046-01ca857be052 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:33.266499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76a9f3c-5f45-4dd1-9046-01ca857be052 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:33.295801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d76a9f3c-5f45-4dd1-9046-01ca857be052 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:33.297017 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d76a9f3c-5f45-4dd1-9046-01ca857be052 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:33.314516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:33.320584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8465f7f-04c0-4a14-a17c-34e517dfe19a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:33.365343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d672d86-e7ef-417e-a53a-4ee0a9b2457a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:33.366171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d672d86-e7ef-417e-a53a-4ee0a9b2457a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:33.366460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d672d86-e7ef-417e-a53a-4ee0a9b2457a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:33.367014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d672d86-e7ef-417e-a53a-4ee0a9b2457a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:33.367802 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d672d86-e7ef-417e-a53a-4ee0a9b2457a None admin] RBAC: Adding query filter params (group.id=2750e7b66ccf40bbb43b081af3edde7b, role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=b8d28080f87c4783b9ef971b31a0d51a) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:33.368161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d672d86-e7ef-417e-a53a-4ee0a9b2457a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:33.369608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d672d86-e7ef-417e-a53a-4ee0a9b2457a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:33.376725 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 176/381] 10.222.0.65 () {56 vars in 1376 bytes} [Wed Aug 24 03:26:33 2022] GET /identity/v3/role_assignments?group.id=2750e7b66ccf40bbb43b081af3edde7b&scope.project.id=b8d28080f87c4783b9ef971b31a0d51a&role.id=43ca25467bc6484da3a50508efb79230 => generated 260 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:33.416629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:33.463892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 206/382] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:33.802674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef2d429-2e8a-4c54-9bff-4cc8b5484b29 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:33.802947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef2d429-2e8a-4c54-9bff-4cc8b5484b29 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:33.803443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef2d429-2e8a-4c54-9bff-4cc8b5484b29 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:33.805225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 177/383] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:33.810400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea247a44-c195-4095-bb48-d725c864e4b9 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:33.810963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea247a44-c195-4095-bb48-d725c864e4b9 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:33.811438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea247a44-c195-4095-bb48-d725c864e4b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:33.836654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-ea247a44-c195-4095-bb48-d725c864e4b9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:33.838230 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ea247a44-c195-4095-bb48-d725c864e4b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:33.921786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 207/384] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:33.924101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef2d429-2e8a-4c54-9bff-4cc8b5484b29 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:34.063554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef2d429-2e8a-4c54-9bff-4cc8b5484b29 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:34.107279 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc66775-d405-4fc6-9173-120686c4d74a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:34.107913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc66775-d405-4fc6-9173-120686c4d74a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:34.108193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc66775-d405-4fc6-9173-120686c4d74a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:34.108463 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc66775-d405-4fc6-9173-120686c4d74a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:34.123870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc66775-d405-4fc6-9173-120686c4d74a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-efc66775-d405-4fc6-9173-120686c4d74a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:34.129392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:34.130373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:34.131457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:34.131457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:34.131457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:34.131457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:34.131457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:34.131457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:34.131457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:34.131457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:34.131457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:34.131457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:34.131457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:34.131457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:34.132559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 178/385] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:26:33 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:34.137225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea247a44-c195-4095-bb48-d725c864e4b9 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:34.245739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc66775-d405-4fc6-9173-120686c4d74a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:34.298846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:34.307824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea247a44-c195-4095-bb48-d725c864e4b9 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:34.333417 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:34.379984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82bda93-35c0-45fd-bab7-7e76519bf23d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:34.380844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82bda93-35c0-45fd-bab7-7e76519bf23d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:34.381132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82bda93-35c0-45fd-bab7-7e76519bf23d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:34.381409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82bda93-35c0-45fd-bab7-7e76519bf23d None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:34.382087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82bda93-35c0-45fd-bab7-7e76519bf23d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:34.382556 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82bda93-35c0-45fd-bab7-7e76519bf23d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:34.383939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82bda93-35c0-45fd-bab7-7e76519bf23d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:34.400149 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 208/386] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:34 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:34.401827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a2fca1-7125-46e8-bc8c-dcb899b3804c None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:34.402117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a2fca1-7125-46e8-bc8c-dcb899b3804c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:34.402446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a2fca1-7125-46e8-bc8c-dcb899b3804c None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:34.404750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 209/387] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:34.410581 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a2fca1-7125-46e8-bc8c-dcb899b3804c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:34.420285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc66775-d405-4fc6-9173-120686c4d74a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:34.467738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259e322a-70a0-42da-bf35-f26fe8e582c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:34.468671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259e322a-70a0-42da-bf35-f26fe8e582c5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:34.470286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259e322a-70a0-42da-bf35-f26fe8e582c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:34.470603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259e322a-70a0-42da-bf35-f26fe8e582c5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:34.489873 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259e322a-70a0-42da-bf35-f26fe8e582c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-259e322a-70a0-42da-bf35-f26fe8e582c5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:34.496332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:34.498390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 179/388] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:34 2022] GET /identity/v3/roles/member => generated 84 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:34.500319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d053c88c-254b-429f-b904-c6f681f00cae None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:34.501939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d053c88c-254b-429f-b904-c6f681f00cae None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:34.501939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d053c88c-254b-429f-b904-c6f681f00cae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:34.527496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d053c88c-254b-429f-b904-c6f681f00cae None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:34.529142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d053c88c-254b-429f-b904-c6f681f00cae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:34.567703 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a2fca1-7125-46e8-bc8c-dcb899b3804c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:34.613628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ccf0f1-3100-48cd-b5f8-80c83956ae58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:34.614493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ccf0f1-3100-48cd-b5f8-80c83956ae58 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:34.614773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ccf0f1-3100-48cd-b5f8-80c83956ae58 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:34.615043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ccf0f1-3100-48cd-b5f8-80c83956ae58 None admin] PATH_INFO: `/v3/groups/f95b8af0f9ae46c7b420273e832ffb67` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:34.615638 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 180/389] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:34.616247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ccf0f1-3100-48cd-b5f8-80c83956ae58 None admin] RBAC: Authorizing `identity:get_group(group_id=f95b8af0f9ae46c7b420273e832ffb67)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:34.616422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d053c88c-254b-429f-b904-c6f681f00cae None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:34.617764 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ccf0f1-3100-48cd-b5f8-80c83956ae58 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:34.624219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 210/390] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:34 2022] GET /identity/v3/groups/f95b8af0f9ae46c7b420273e832ffb67 => generated 226 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:34.626982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ccf0f1-3100-48cd-b5f8-80c83956ae58 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:34.632880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:34.674210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:34.740891 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ccf0f1-3100-48cd-b5f8-80c83956ae58 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:34.781991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d053c88c-254b-429f-b904-c6f681f00cae None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:34.787566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65970dee-95bd-4ef8-9a32-f1e2400b6d7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:34.788249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65970dee-95bd-4ef8-9a32-f1e2400b6d7a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:34.791257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65970dee-95bd-4ef8-9a32-f1e2400b6d7a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:34.791535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65970dee-95bd-4ef8-9a32-f1e2400b6d7a None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:34.794465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65970dee-95bd-4ef8-9a32-f1e2400b6d7a None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:34.795996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65970dee-95bd-4ef8-9a32-f1e2400b6d7a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:34.799146 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 211/391] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:34 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:34.803744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65970dee-95bd-4ef8-9a32-f1e2400b6d7a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:34.850594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa55a7d-05c5-44b2-af8f-b5b68a72fe78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:34.851212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa55a7d-05c5-44b2-af8f-b5b68a72fe78 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:34.851491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa55a7d-05c5-44b2-af8f-b5b68a72fe78 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:34.851763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa55a7d-05c5-44b2-af8f-b5b68a72fe78 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:34.852431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa55a7d-05c5-44b2-af8f-b5b68a72fe78 None admin] RBAC: Adding query filter params (name=member) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:34.852731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa55a7d-05c5-44b2-af8f-b5b68a72fe78 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:34.856053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa55a7d-05c5-44b2-af8f-b5b68a72fe78 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:34.884245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 181/392] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:34.889702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa55a7d-05c5-44b2-af8f-b5b68a72fe78 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:34.948227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65970dee-95bd-4ef8-9a32-f1e2400b6d7a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:34.955863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:34.987950 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c5d555-e79b-40a4-8fcf-77716b774fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:34.988936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c5d555-e79b-40a4-8fcf-77716b774fc5 None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:34.990256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c5d555-e79b-40a4-8fcf-77716b774fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:34.990732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c5d555-e79b-40a4-8fcf-77716b774fc5 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db/groups/f95b8af0f9ae46c7b420273e832ffb67/roles/d02a8e6c49b74aaa9403f8bd535b90bf` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:34.993163 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c5d555-e79b-40a4-8fcf-77716b774fc5 None admin] RBAC: Authorizing `identity:create_grant(project_id=db44137c64a347cfb350dad24d36a3db, group_id=f95b8af0f9ae46c7b420273e832ffb67, role_id=d02a8e6c49b74aaa9403f8bd535b90bf)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:34.994810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c5d555-e79b-40a4-8fcf-77716b774fc5 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:35.000715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:35.020568 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa55a7d-05c5-44b2-af8f-b5b68a72fe78 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:35.053102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac66058-32a4-4fab-9184-a81721a0ac12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:35.053725 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac66058-32a4-4fab-9184-a81721a0ac12 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:35.053999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac66058-32a4-4fab-9184-a81721a0ac12 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:35.054293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac66058-32a4-4fab-9184-a81721a0ac12 None admin] PATH_INFO: `/v3/groups/f95b8af0f9ae46c7b420273e832ffb67` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:35.055414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac66058-32a4-4fab-9184-a81721a0ac12 None admin] RBAC: Authorizing `identity:get_group(group_id=f95b8af0f9ae46c7b420273e832ffb67)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:35.056832 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac66058-32a4-4fab-9184-a81721a0ac12 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:35.058607 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 182/393] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:34 2022] GET /identity/v3/groups/f95b8af0f9ae46c7b420273e832ffb67 => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:35.063402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac66058-32a4-4fab-9184-a81721a0ac12 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:35.073095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 212/394] 10.222.0.65 () {58 vars in 1442 bytes} [Wed Aug 24 03:26:34 2022] PUT /identity/v3/projects/db44137c64a347cfb350dad24d36a3db/groups/f95b8af0f9ae46c7b420273e832ffb67/roles/d02a8e6c49b74aaa9403f8bd535b90bf => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:35.074833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c5d555-e79b-40a4-8fcf-77716b774fc5 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:35.221782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c5d555-e79b-40a4-8fcf-77716b774fc5 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:35.225661 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac66058-32a4-4fab-9184-a81721a0ac12 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:35.265557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329ae4ad-88cc-4d9b-9295-8c14ade84206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:35.266596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329ae4ad-88cc-4d9b-9295-8c14ade84206 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:35.267099 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329ae4ad-88cc-4d9b-9295-8c14ade84206 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:35.267568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329ae4ad-88cc-4d9b-9295-8c14ade84206 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:35.269709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329ae4ad-88cc-4d9b-9295-8c14ade84206 None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:35.272066 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329ae4ad-88cc-4d9b-9295-8c14ade84206 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:35.279069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 213/395] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:35 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:35.281188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329ae4ad-88cc-4d9b-9295-8c14ade84206 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:35.317213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8884228-8835-4026-8770-61b855671b1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:35.318139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8884228-8835-4026-8770-61b855671b1b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:35.318446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8884228-8835-4026-8770-61b855671b1b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:35.318717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8884228-8835-4026-8770-61b855671b1b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:35.324694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:35.332781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8884228-8835-4026-8770-61b855671b1b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-f8884228-8835-4026-8770-61b855671b1b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:35.339806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:35.340828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:35.341775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 183/396] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:35.346600 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8884228-8835-4026-8770-61b855671b1b None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:35.351643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:35.418640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329ae4ad-88cc-4d9b-9295-8c14ade84206 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:35.465950 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326f57cc-09af-4097-a6ea-236da8e7ba49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:35.466866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326f57cc-09af-4097-a6ea-236da8e7ba49 None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:35.467235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326f57cc-09af-4097-a6ea-236da8e7ba49 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:35.467614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326f57cc-09af-4097-a6ea-236da8e7ba49 None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db/groups/f95b8af0f9ae46c7b420273e832ffb67/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:35.469732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326f57cc-09af-4097-a6ea-236da8e7ba49 None admin] RBAC: Authorizing `identity:create_grant(project_id=db44137c64a347cfb350dad24d36a3db, group_id=f95b8af0f9ae46c7b420273e832ffb67, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:35.471440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326f57cc-09af-4097-a6ea-236da8e7ba49 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:35.480104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8884228-8835-4026-8770-61b855671b1b None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:35.517073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38bc69a-d920-421c-b64e-f59b6d260dd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:35.517735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38bc69a-d920-421c-b64e-f59b6d260dd3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:35.518004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38bc69a-d920-421c-b64e-f59b6d260dd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:35.518294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38bc69a-d920-421c-b64e-f59b6d260dd3 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:35.519641 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38bc69a-d920-421c-b64e-f59b6d260dd3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:35.520164 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38bc69a-d920-421c-b64e-f59b6d260dd3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:35.521598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38bc69a-d920-421c-b64e-f59b6d260dd3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:35.533598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 184/397] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:35.535354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc54925a-d81f-4d38-96d7-d316bff22d5e None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:35.535846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc54925a-d81f-4d38-96d7-d316bff22d5e None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:35.536189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc54925a-d81f-4d38-96d7-d316bff22d5e None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:35.537617 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 185/398] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:35.538748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc54925a-d81f-4d38-96d7-d316bff22d5e None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:35.560286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 214/399] 10.222.0.65 () {58 vars in 1442 bytes} [Wed Aug 24 03:26:35 2022] PUT /identity/v3/projects/db44137c64a347cfb350dad24d36a3db/groups/f95b8af0f9ae46c7b420273e832ffb67/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:35.561713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe96207b-0acf-49d2-bd46-e6e25b160eb4 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:35.562071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe96207b-0acf-49d2-bd46-e6e25b160eb4 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:35.562451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe96207b-0acf-49d2-bd46-e6e25b160eb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:35.590610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-fe96207b-0acf-49d2-bd46-e6e25b160eb4 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:35.592498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-fe96207b-0acf-49d2-bd46-e6e25b160eb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:35.615325 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:35.661516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc54925a-d81f-4d38-96d7-d316bff22d5e None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:35.690101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 215/400] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:35.701810 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51405ed-686b-4859-85e2-381ed04da980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:35.702735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51405ed-686b-4859-85e2-381ed04da980 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:35.703240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51405ed-686b-4859-85e2-381ed04da980 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:35.703684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51405ed-686b-4859-85e2-381ed04da980 None admin] PATH_INFO: `/v3/groups/f95b8af0f9ae46c7b420273e832ffb67` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:35.704990 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51405ed-686b-4859-85e2-381ed04da980 None admin] RBAC: Authorizing `identity:get_group(group_id=f95b8af0f9ae46c7b420273e832ffb67)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:35.707523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51405ed-686b-4859-85e2-381ed04da980 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:35.709514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 186/401] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:35 2022] GET /identity/v3/groups/f95b8af0f9ae46c7b420273e832ffb67 => generated 226 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:35.715636 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe96207b-0acf-49d2-bd46-e6e25b160eb4 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:35.757395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:35.842329 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe96207b-0acf-49d2-bd46-e6e25b160eb4 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:35.879087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8730cc-404d-41a5-9de4-ee7378db0a16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:35.879825 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8730cc-404d-41a5-9de4-ee7378db0a16 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:35.880105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8730cc-404d-41a5-9de4-ee7378db0a16 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:35.880378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8730cc-404d-41a5-9de4-ee7378db0a16 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:35.881767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8730cc-404d-41a5-9de4-ee7378db0a16 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:35.883363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8730cc-404d-41a5-9de4-ee7378db0a16 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:35.885330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 216/402] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:35 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:35.890561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51405ed-686b-4859-85e2-381ed04da980 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:35.929913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95cd7b1-8285-46b8-b8a7-26617f72ee3f None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:35.930263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95cd7b1-8285-46b8-b8a7-26617f72ee3f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:35.930494 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95cd7b1-8285-46b8-b8a7-26617f72ee3f None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:35.931826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 217/403] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:26:35.941311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4348c1-f301-4f5b-8118-3e28b52ce387 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:35.941683 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4348c1-f301-4f5b-8118-3e28b52ce387 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:35.941901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4348c1-f301-4f5b-8118-3e28b52ce387 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:35.975495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-0d4348c1-f301-4f5b-8118-3e28b52ce387 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:35.978247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0d4348c1-f301-4f5b-8118-3e28b52ce387 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:36.068331 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51405ed-686b-4859-85e2-381ed04da980 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:36.092732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 218/404] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:36.098982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4348c1-f301-4f5b-8118-3e28b52ce387 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:36.116137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:36.125083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4235ec-854a-4100-bd7e-ac31f841c62a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:36.131476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4235ec-854a-4100-bd7e-ac31f841c62a None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:36.131476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4235ec-854a-4100-bd7e-ac31f841c62a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:36.131476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4235ec-854a-4100-bd7e-ac31f841c62a None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3/groups/f95b8af0f9ae46c7b420273e832ffb67/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:36.131476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4235ec-854a-4100-bd7e-ac31f841c62a None admin] RBAC: Authorizing `identity:create_grant(project_id=5c50437619304d0d857f65feed1d19e3, group_id=f95b8af0f9ae46c7b420273e832ffb67, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:36.132562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4235ec-854a-4100-bd7e-ac31f841c62a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:36.144598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:36.226070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4348c1-f301-4f5b-8118-3e28b52ce387 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:36.248003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 187/405] 10.222.0.65 () {58 vars in 1442 bytes} [Wed Aug 24 03:26:35 2022] PUT /identity/v3/projects/5c50437619304d0d857f65feed1d19e3/groups/f95b8af0f9ae46c7b420273e832ffb67/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:36.280059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35ccdd7-c99d-45c8-8fb3-fce82115bae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:36.280902 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35ccdd7-c99d-45c8-8fb3-fce82115bae1 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:36.281196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35ccdd7-c99d-45c8-8fb3-fce82115bae1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:36.281479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35ccdd7-c99d-45c8-8fb3-fce82115bae1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:36.297545 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35ccdd7-c99d-45c8-8fb3-fce82115bae1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-a35ccdd7-c99d-45c8-8fb3-fce82115bae1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:36.303059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:36.303957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:36.304723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 219/406] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:26:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:36.310563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4235ec-854a-4100-bd7e-ac31f841c62a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:36.430083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4235ec-854a-4100-bd7e-ac31f841c62a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:36.466334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52459a48-5dca-4c7b-8efc-eabbe99b0ef8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:36.466947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52459a48-5dca-4c7b-8efc-eabbe99b0ef8 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:36.467223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52459a48-5dca-4c7b-8efc-eabbe99b0ef8 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:36.467489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52459a48-5dca-4c7b-8efc-eabbe99b0ef8 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:36.468156 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52459a48-5dca-4c7b-8efc-eabbe99b0ef8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:36.468455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52459a48-5dca-4c7b-8efc-eabbe99b0ef8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:36.469818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52459a48-5dca-4c7b-8efc-eabbe99b0ef8 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:36.479955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 188/407] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:36.485365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35ccdd7-c99d-45c8-8fb3-fce82115bae1 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:36.500975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52459a48-5dca-4c7b-8efc-eabbe99b0ef8 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:36.574804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:36.587677 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:36.637309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52459a48-5dca-4c7b-8efc-eabbe99b0ef8 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:36.641917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35ccdd7-c99d-45c8-8fb3-fce82115bae1 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:36.701494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e3e60d-710d-4fb1-803e-ae8a786dc934 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:36.702045 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e3e60d-710d-4fb1-803e-ae8a786dc934 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:36.702345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e3e60d-710d-4fb1-803e-ae8a786dc934 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:36.702612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e3e60d-710d-4fb1-803e-ae8a786dc934 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:36.712064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cba2cb4-6f6a-413a-8f9f-ad7c72d33e98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:36.712692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cba2cb4-6f6a-413a-8f9f-ad7c72d33e98 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:36.712983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cba2cb4-6f6a-413a-8f9f-ad7c72d33e98 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:36.713232 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cba2cb4-6f6a-413a-8f9f-ad7c72d33e98 None admin] PATH_INFO: `/v3/groups/f95b8af0f9ae46c7b420273e832ffb67` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:36.716115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cba2cb4-6f6a-413a-8f9f-ad7c72d33e98 None admin] RBAC: Authorizing `identity:get_group(group_id=f95b8af0f9ae46c7b420273e832ffb67)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:36.718443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cba2cb4-6f6a-413a-8f9f-ad7c72d33e98 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:36.724195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 220/408] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:36 2022] GET /identity/v3/groups/f95b8af0f9ae46c7b420273e832ffb67 => generated 226 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:36.727100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e3e60d-710d-4fb1-803e-ae8a786dc934 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-a6e3e60d-710d-4fb1-803e-ae8a786dc934 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:36.734355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:36.735242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:36.736039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:36.736039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:36.736039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:36.736039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:36.736039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:36.736039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:36.736039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:36.736039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:36.736039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:36.736039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:36.736039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:36.736039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:36.736642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cba2cb4-6f6a-413a-8f9f-ad7c72d33e98 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:36.744579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 189/409] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:26:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:36.746392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e3e60d-710d-4fb1-803e-ae8a786dc934 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:36.898614 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e3e60d-710d-4fb1-803e-ae8a786dc934 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:36.914354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:36.940584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f965ba-c3ff-44d3-ab89-86cb42e69753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:36.941835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f965ba-c3ff-44d3-ab89-86cb42e69753 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:36.942429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f965ba-c3ff-44d3-ab89-86cb42e69753 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:36.943150 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f965ba-c3ff-44d3-ab89-86cb42e69753 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:36.947121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f965ba-c3ff-44d3-ab89-86cb42e69753 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:36.948369 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f965ba-c3ff-44d3-ab89-86cb42e69753 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:36.950166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f965ba-c3ff-44d3-ab89-86cb42e69753 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:36.960582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cba2cb4-6f6a-413a-8f9f-ad7c72d33e98 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:36.969851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 190/410] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:36.976379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f965ba-c3ff-44d3-ab89-86cb42e69753 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:37.017995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050585f2-11f9-4136-bd3d-767d48e703fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:37.018692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050585f2-11f9-4136-bd3d-767d48e703fa None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:37.019647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050585f2-11f9-4136-bd3d-767d48e703fa None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:37.020081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050585f2-11f9-4136-bd3d-767d48e703fa None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:37.027121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050585f2-11f9-4136-bd3d-767d48e703fa None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:37.028602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050585f2-11f9-4136-bd3d-767d48e703fa None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:37.030719 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 221/411] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:36 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:37.036536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050585f2-11f9-4136-bd3d-767d48e703fa None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:37.083537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:37.093042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f965ba-c3ff-44d3-ab89-86cb42e69753 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:37.138181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52a9f58-99af-42c4-ba22-f13920a83253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:37.138181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52a9f58-99af-42c4-ba22-f13920a83253 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:37.138181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52a9f58-99af-42c4-ba22-f13920a83253 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:37.138181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52a9f58-99af-42c4-ba22-f13920a83253 None admin] PATH_INFO: `/v3/groups/2750e7b66ccf40bbb43b081af3edde7b` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:37.138602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52a9f58-99af-42c4-ba22-f13920a83253 None admin] RBAC: Authorizing `identity:get_group(group_id=2750e7b66ccf40bbb43b081af3edde7b)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:37.138602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52a9f58-99af-42c4-ba22-f13920a83253 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:37.140139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 191/412] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:36 2022] GET /identity/v3/groups/2750e7b66ccf40bbb43b081af3edde7b => generated 229 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:37.146760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52a9f58-99af-42c4-ba22-f13920a83253 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:37.191409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050585f2-11f9-4136-bd3d-767d48e703fa None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:37.220382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:37.240228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14cbcb33-443e-4ce8-abd7-038cdf505288 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:37.246271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cbcb33-443e-4ce8-abd7-038cdf505288 None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:37.246271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cbcb33-443e-4ce8-abd7-038cdf505288 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:37.246271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cbcb33-443e-4ce8-abd7-038cdf505288 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3/groups/f95b8af0f9ae46c7b420273e832ffb67/roles/d02a8e6c49b74aaa9403f8bd535b90bf` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:37.248332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14cbcb33-443e-4ce8-abd7-038cdf505288 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c50437619304d0d857f65feed1d19e3, group_id=f95b8af0f9ae46c7b420273e832ffb67, role_id=d02a8e6c49b74aaa9403f8bd535b90bf)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:37.249948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14cbcb33-443e-4ce8-abd7-038cdf505288 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:37.285936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52a9f58-99af-42c4-ba22-f13920a83253 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:37.326916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246dfa4e-855f-4561-a088-a845a3eb9fc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:37.327411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 222/413] 10.222.0.65 () {58 vars in 1442 bytes} [Wed Aug 24 03:26:37 2022] PUT /identity/v3/projects/5c50437619304d0d857f65feed1d19e3/groups/f95b8af0f9ae46c7b420273e832ffb67/roles/d02a8e6c49b74aaa9403f8bd535b90bf => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:26:37.328133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246dfa4e-855f-4561-a088-a845a3eb9fc9 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:37.328588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246dfa4e-855f-4561-a088-a845a3eb9fc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:37.329047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246dfa4e-855f-4561-a088-a845a3eb9fc9 None admin] PATH_INFO: `/v3/projects/b8d28080f87c4783b9ef971b31a0d51a` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:37.330619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246dfa4e-855f-4561-a088-a845a3eb9fc9 None admin] RBAC: Authorizing `identity:get_project(project_id=b8d28080f87c4783b9ef971b31a0d51a)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:37.332535 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246dfa4e-855f-4561-a088-a845a3eb9fc9 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:37.334820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788f173d-8550-49b8-a8b1-8da4096784a9 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:37.335153 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 192/414] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:37 2022] GET /identity/v3/projects/b8d28080f87c4783b9ef971b31a0d51a => generated 344 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:37.341149 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788f173d-8550-49b8-a8b1-8da4096784a9 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:37.341149 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788f173d-8550-49b8-a8b1-8da4096784a9 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:37.341149 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 223/415] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:37 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:37.342892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfdcf59-22a0-454f-936b-9854b3c301c2 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:37.343371 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788f173d-8550-49b8-a8b1-8da4096784a9 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:37.343537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfdcf59-22a0-454f-936b-9854b3c301c2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:37.344053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfdcf59-22a0-454f-936b-9854b3c301c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:37.368523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-5dfdcf59-22a0-454f-936b-9854b3c301c2 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:37.370256 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5dfdcf59-22a0-454f-936b-9854b3c301c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:37.460497 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 193/416] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:37.530650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:37.567723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788f173d-8550-49b8-a8b1-8da4096784a9 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:37.611520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361d8ac6-2a32-4312-a883-4522b5a6a425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:37.612186 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361d8ac6-2a32-4312-a883-4522b5a6a425 None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:37.612468 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361d8ac6-2a32-4312-a883-4522b5a6a425 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:37.613360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361d8ac6-2a32-4312-a883-4522b5a6a425 None admin] PATH_INFO: `/v3/projects/b8d28080f87c4783b9ef971b31a0d51a/groups/2750e7b66ccf40bbb43b081af3edde7b/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:37.619560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361d8ac6-2a32-4312-a883-4522b5a6a425 None admin] RBAC: Authorizing `identity:create_grant(project_id=b8d28080f87c4783b9ef971b31a0d51a, group_id=2750e7b66ccf40bbb43b081af3edde7b, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:37.621312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361d8ac6-2a32-4312-a883-4522b5a6a425 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:37.695702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 224/417] 10.222.0.65 () {58 vars in 1442 bytes} [Wed Aug 24 03:26:37 2022] PUT /identity/v3/projects/b8d28080f87c4783b9ef971b31a0d51a/groups/2750e7b66ccf40bbb43b081af3edde7b/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:37.821314 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfdcf59-22a0-454f-936b-9854b3c301c2 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:37.876906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:37.934125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e1e911-8246-45a9-bbb8-848056e68fc3 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:37.934486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e1e911-8246-45a9-bbb8-848056e68fc3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:37.934716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e1e911-8246-45a9-bbb8-848056e68fc3 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:37.936198 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 225/418] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:37.941873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b2567d-d0d6-4516-8ee0-f2fed7f07c8b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:37.942187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b2567d-d0d6-4516-8ee0-f2fed7f07c8b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:37.942466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b2567d-d0d6-4516-8ee0-f2fed7f07c8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:37.949264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfdcf59-22a0-454f-936b-9854b3c301c2 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:37.967349 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c2b2567d-d0d6-4516-8ee0-f2fed7f07c8b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:37.968885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c2b2567d-d0d6-4516-8ee0-f2fed7f07c8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:37.994493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb19111-6493-4fa6-9011-bb0f3b008828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:37.995168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb19111-6493-4fa6-9011-bb0f3b008828 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:37.995448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb19111-6493-4fa6-9011-bb0f3b008828 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:37.995715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb19111-6493-4fa6-9011-bb0f3b008828 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:38.009758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb19111-6493-4fa6-9011-bb0f3b008828 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-4bb19111-6493-4fa6-9011-bb0f3b008828 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:38.015892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:38.016763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:38.017562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 194/419] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:26:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:38.028281 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb19111-6493-4fa6-9011-bb0f3b008828 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:38.079330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 226/420] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:26:38.196545 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb19111-6493-4fa6-9011-bb0f3b008828 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:38.198206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:38.238331 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0dd1000-9e3f-4425-9c17-c3b6d5fa1b94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:38.239423 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0dd1000-9e3f-4425-9c17-c3b6d5fa1b94 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:38.239895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0dd1000-9e3f-4425-9c17-c3b6d5fa1b94 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:38.240470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0dd1000-9e3f-4425-9c17-c3b6d5fa1b94 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:38.241346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0dd1000-9e3f-4425-9c17-c3b6d5fa1b94 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:38.246629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0dd1000-9e3f-4425-9c17-c3b6d5fa1b94 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:38.248260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0dd1000-9e3f-4425-9c17-c3b6d5fa1b94 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:38.258327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 195/421] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:38.263944 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b2567d-d0d6-4516-8ee0-f2fed7f07c8b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:38.352100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:38.387541 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b2567d-d0d6-4516-8ee0-f2fed7f07c8b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:38.429064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdc9bc4-adb9-4a3b-bfa9-bddea97e66dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:38.429687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdc9bc4-adb9-4a3b-bfa9-bddea97e66dd None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:38.429971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdc9bc4-adb9-4a3b-bfa9-bddea97e66dd None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:38.430270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdc9bc4-adb9-4a3b-bfa9-bddea97e66dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:38.497840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0dd1000-9e3f-4425-9c17-c3b6d5fa1b94 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:38.501774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cdc9bc4-adb9-4a3b-bfa9-bddea97e66dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:38.504348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cdc9bc4-adb9-4a3b-bfa9-bddea97e66dd None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:38.572382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:38.612629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 227/422] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:38.613959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdc9bc4-adb9-4a3b-bfa9-bddea97e66dd None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:38.718227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0dd1000-9e3f-4425-9c17-c3b6d5fa1b94 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:38.782675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdc9bc4-adb9-4a3b-bfa9-bddea97e66dd None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:38.792933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442539b6-965d-4d30-ba07-0f10c0e5ef4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:38.793668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442539b6-965d-4d30-ba07-0f10c0e5ef4e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:38.793944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442539b6-965d-4d30-ba07-0f10c0e5ef4e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:38.794246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442539b6-965d-4d30-ba07-0f10c0e5ef4e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:38.805773 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442539b6-965d-4d30-ba07-0f10c0e5ef4e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-442539b6-965d-4d30-ba07-0f10c0e5ef4e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:38.811724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:38.812717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:38.813544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 196/423] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:38.814753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7c0db7-e784-4ddd-8c38-ffdfe257e6e1 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:38.815013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7c0db7-e784-4ddd-8c38-ffdfe257e6e1 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:38.815293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7c0db7-e784-4ddd-8c38-ffdfe257e6e1 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:38.820118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483438da-392a-478e-b8dd-6d6b8c371c2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:38.820780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483438da-392a-478e-b8dd-6d6b8c371c2a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:38.821059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483438da-392a-478e-b8dd-6d6b8c371c2a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:38.821334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483438da-392a-478e-b8dd-6d6b8c371c2a None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:38.821977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 197/424] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:38.822763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483438da-392a-478e-b8dd-6d6b8c371c2a None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:38.823425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14312efd-5122-4900-b3a3-44280b3640f2 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:38.823707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14312efd-5122-4900-b3a3-44280b3640f2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:38.823978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14312efd-5122-4900-b3a3-44280b3640f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:38.824244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483438da-392a-478e-b8dd-6d6b8c371c2a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:38.826255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 228/425] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:38 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:38.827287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483438da-392a-478e-b8dd-6d6b8c371c2a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:38.843326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-14312efd-5122-4900-b3a3-44280b3640f2 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:38.844629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-14312efd-5122-4900-b3a3-44280b3640f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:38.863538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:38.880489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:38.922610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 198/426] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:38.923737 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14312efd-5122-4900-b3a3-44280b3640f2 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:38.932485 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483438da-392a-478e-b8dd-6d6b8c371c2a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:38.982869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbfec17-1386-4332-a495-5252729ecea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:38.983487 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbfec17-1386-4332-a495-5252729ecea8 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:38.983770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbfec17-1386-4332-a495-5252729ecea8 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:38.984039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbfec17-1386-4332-a495-5252729ecea8 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:38.984706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbfec17-1386-4332-a495-5252729ecea8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:38.985004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbfec17-1386-4332-a495-5252729ecea8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:38.986394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbfec17-1386-4332-a495-5252729ecea8 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:39.002863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 229/427] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:39.008267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbfec17-1386-4332-a495-5252729ecea8 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:39.093815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14312efd-5122-4900-b3a3-44280b3640f2 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:39.139578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca9ea14-0d62-430a-8a4e-01087e996977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:39.140174 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca9ea14-0d62-430a-8a4e-01087e996977 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:39.140498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca9ea14-0d62-430a-8a4e-01087e996977 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:39.140767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca9ea14-0d62-430a-8a4e-01087e996977 None admin] PATH_INFO: `/v3/groups/f95b8af0f9ae46c7b420273e832ffb67` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:39.142374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca9ea14-0d62-430a-8a4e-01087e996977 None admin] RBAC: Authorizing `identity:get_group(group_id=f95b8af0f9ae46c7b420273e832ffb67)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:39.143326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca9ea14-0d62-430a-8a4e-01087e996977 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:39.145736 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 199/428] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:38 2022] GET /identity/v3/groups/f95b8af0f9ae46c7b420273e832ffb67 => generated 226 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:39.151343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca9ea14-0d62-430a-8a4e-01087e996977 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:39.189233 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:39.193992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:39.196037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbfec17-1386-4332-a495-5252729ecea8 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:39.257548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11e3851-0d44-4c0f-be00-ea2144cd0dea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:39.258575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11e3851-0d44-4c0f-be00-ea2144cd0dea None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:39.259053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11e3851-0d44-4c0f-be00-ea2144cd0dea None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:39.259535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11e3851-0d44-4c0f-be00-ea2144cd0dea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:39.313660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca9ea14-0d62-430a-8a4e-01087e996977 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:39.337618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11e3851-0d44-4c0f-be00-ea2144cd0dea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:39.339240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11e3851-0d44-4c0f-be00-ea2144cd0dea None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:39.353601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dcea0c-10e5-4aa9-83e5-2a626b448fe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:39.354489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dcea0c-10e5-4aa9-83e5-2a626b448fe3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:39.355003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dcea0c-10e5-4aa9-83e5-2a626b448fe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:39.355447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dcea0c-10e5-4aa9-83e5-2a626b448fe3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:39.356652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dcea0c-10e5-4aa9-83e5-2a626b448fe3 None admin] RBAC: Adding query filter params (group.id=f95b8af0f9ae46c7b420273e832ffb67, role.id=d02a8e6c49b74aaa9403f8bd535b90bf, scope.project.id=db44137c64a347cfb350dad24d36a3db) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:39.357165 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dcea0c-10e5-4aa9-83e5-2a626b448fe3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:39.358990 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57dcea0c-10e5-4aa9-83e5-2a626b448fe3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:39.366091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 200/429] 10.222.0.65 () {56 vars in 1376 bytes} [Wed Aug 24 03:26:39 2022] GET /identity/v3/role_assignments?group.id=f95b8af0f9ae46c7b420273e832ffb67&scope.project.id=db44137c64a347cfb350dad24d36a3db&role.id=d02a8e6c49b74aaa9403f8bd535b90bf => generated 611 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:39.367218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dcea0c-10e5-4aa9-83e5-2a626b448fe3 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:39.451419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:39.456271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 230/430] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 03:26:39.467410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11e3851-0d44-4c0f-be00-ea2144cd0dea None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:39.468576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:39.485928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dcea0c-10e5-4aa9-83e5-2a626b448fe3 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:39.531024 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b605ed6-162e-4221-91a9-6e0db02690a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:39.532120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b605ed6-162e-4221-91a9-6e0db02690a6 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:39.532609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b605ed6-162e-4221-91a9-6e0db02690a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:39.533100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b605ed6-162e-4221-91a9-6e0db02690a6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:39.556200 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b605ed6-162e-4221-91a9-6e0db02690a6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-3b605ed6-162e-4221-91a9-6e0db02690a6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:39.571142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:39.572101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:39.572907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:39.572907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:39.572907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:39.572907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:39.572907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:39.572907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:39.572907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:39.572907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:39.572907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:39.572907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:39.572907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 24 03:26:39.572907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:39.575402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 201/431] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:39.583310 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b605ed6-162e-4221-91a9-6e0db02690a6 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:39.598182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11e3851-0d44-4c0f-be00-ea2144cd0dea None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:39.631597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8c2122-ab76-4abd-bb99-6d2afafccf4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:39.632252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8c2122-ab76-4abd-bb99-6d2afafccf4c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:39.632593 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8c2122-ab76-4abd-bb99-6d2afafccf4c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:39.632873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8c2122-ab76-4abd-bb99-6d2afafccf4c None admin] PATH_INFO: `/v3/projects/db44137c64a347cfb350dad24d36a3db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:39.634173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8c2122-ab76-4abd-bb99-6d2afafccf4c None admin] RBAC: Authorizing `identity:get_project(project_id=db44137c64a347cfb350dad24d36a3db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:39.635877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8c2122-ab76-4abd-bb99-6d2afafccf4c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:39.637881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 231/432] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:39 2022] GET /identity/v3/projects/db44137c64a347cfb350dad24d36a3db => generated 298 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:39.658806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8c2122-ab76-4abd-bb99-6d2afafccf4c None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:39.730856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b605ed6-162e-4221-91a9-6e0db02690a6 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:39.739995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:39.776798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7677e203-93ea-45e4-b981-50b695fee20b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:39.777405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7677e203-93ea-45e4-b981-50b695fee20b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:39.777680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7677e203-93ea-45e4-b981-50b695fee20b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:39.777956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7677e203-93ea-45e4-b981-50b695fee20b None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:39.778662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7677e203-93ea-45e4-b981-50b695fee20b None admin] RBAC: Adding query filter params (name=member) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:39.778959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7677e203-93ea-45e4-b981-50b695fee20b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:39.780234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8c2122-ab76-4abd-bb99-6d2afafccf4c None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:39.780343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7677e203-93ea-45e4-b981-50b695fee20b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:39.790884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 202/433] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:26:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:39.792178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27370981-fa90-4ac9-8c5f-5b2caf9337ae None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:39.796769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27370981-fa90-4ac9-8c5f-5b2caf9337ae None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:39.796769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27370981-fa90-4ac9-8c5f-5b2caf9337ae None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:39.796769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 203/434] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:39.798003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27370981-fa90-4ac9-8c5f-5b2caf9337ae None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:39.810922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:39.825167 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af461e9a-1150-4523-9c14-6371928388f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:39.826281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af461e9a-1150-4523-9c14-6371928388f3 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:39.826818 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af461e9a-1150-4523-9c14-6371928388f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:39.827098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af461e9a-1150-4523-9c14-6371928388f3 None admin] PATH_INFO: `/v3/groups/f95b8af0f9ae46c7b420273e832ffb67` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:39.828225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af461e9a-1150-4523-9c14-6371928388f3 None admin] RBAC: Authorizing `identity:get_group(group_id=f95b8af0f9ae46c7b420273e832ffb67)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:39.829755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af461e9a-1150-4523-9c14-6371928388f3 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:39.831595 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 232/435] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:39 2022] GET /identity/v3/groups/f95b8af0f9ae46c7b420273e832ffb67 => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:39.832828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1f6f5f-3499-42b5-b343-29642f734f65 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:39.833016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1f6f5f-3499-42b5-b343-29642f734f65 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:39.833276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1f6f5f-3499-42b5-b343-29642f734f65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:39.851122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-8a1f6f5f-3499-42b5-b343-29642f734f65 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:39.852314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8a1f6f5f-3499-42b5-b343-29642f734f65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:39.914958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27370981-fa90-4ac9-8c5f-5b2caf9337ae None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:39.930175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 233/436] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:39.930175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1f6f5f-3499-42b5-b343-29642f734f65 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:39.958437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fdb0da-4eba-425d-94c8-1f0857e7f726 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:39.959171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fdb0da-4eba-425d-94c8-1f0857e7f726 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:39.959571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fdb0da-4eba-425d-94c8-1f0857e7f726 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:39.959913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fdb0da-4eba-425d-94c8-1f0857e7f726 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:40.009077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:40.033338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fdb0da-4eba-425d-94c8-1f0857e7f726 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:40.036407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fdb0da-4eba-425d-94c8-1f0857e7f726 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:40.047373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:40.056776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1f6f5f-3499-42b5-b343-29642f734f65 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:40.090834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25595a58-bd6d-43e6-a411-a828125c4713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:40.091442 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25595a58-bd6d-43e6-a411-a828125c4713 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:40.091722 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25595a58-bd6d-43e6-a411-a828125c4713 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:40.091996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25595a58-bd6d-43e6-a411-a828125c4713 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:40.092717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25595a58-bd6d-43e6-a411-a828125c4713 None admin] RBAC: Adding query filter params (group.id=f95b8af0f9ae46c7b420273e832ffb67, role.id=9912acfb698e41ebbce6a5147fafe896, scope.project.id=db44137c64a347cfb350dad24d36a3db) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:40.093022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25595a58-bd6d-43e6-a411-a828125c4713 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:40.094499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25595a58-bd6d-43e6-a411-a828125c4713 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:40.102321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 234/437] 10.222.0.65 () {56 vars in 1376 bytes} [Wed Aug 24 03:26:39 2022] GET /identity/v3/role_assignments?group.id=f95b8af0f9ae46c7b420273e832ffb67&scope.project.id=db44137c64a347cfb350dad24d36a3db&role.id=9912acfb698e41ebbce6a5147fafe896 => generated 611 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:40.102321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8daf21-02ba-4d73-87ad-0244201970fa None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:40.102321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8daf21-02ba-4d73-87ad-0244201970fa None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:40.102595 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8daf21-02ba-4d73-87ad-0244201970fa None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:40.104122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 235/438] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:40 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:40.110286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611e4f13-da35-4800-a40d-871c4cd7db36 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:40.110568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611e4f13-da35-4800-a40d-871c4cd7db36 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:40.110902 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611e4f13-da35-4800-a40d-871c4cd7db36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:40.134950 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-611e4f13-da35-4800-a40d-871c4cd7db36 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:40.136920 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-611e4f13-da35-4800-a40d-871c4cd7db36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:40.154332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 204/439] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:40.162854 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fdb0da-4eba-425d-94c8-1f0857e7f726 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:40.226612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 236/440] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:26:40.227861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611e4f13-da35-4800-a40d-871c4cd7db36 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:40.296567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fdb0da-4eba-425d-94c8-1f0857e7f726 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:40.301128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:40.308915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:40.344397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e395fed9-a88a-41f5-8db6-5fc228daf1f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:40.345355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e395fed9-a88a-41f5-8db6-5fc228daf1f5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:40.345877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e395fed9-a88a-41f5-8db6-5fc228daf1f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:40.346343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e395fed9-a88a-41f5-8db6-5fc228daf1f5 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:40.349490 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611e4f13-da35-4800-a40d-871c4cd7db36 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:40.349899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e395fed9-a88a-41f5-8db6-5fc228daf1f5 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:40.351782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e395fed9-a88a-41f5-8db6-5fc228daf1f5 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:40.354320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 205/441] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:40 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:40.359487 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e395fed9-a88a-41f5-8db6-5fc228daf1f5 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:40.392755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38df9b5c-2de8-4b29-8596-24d7c0a047ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:40.392976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38df9b5c-2de8-4b29-8596-24d7c0a047ad None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:40.393220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38df9b5c-2de8-4b29-8596-24d7c0a047ad None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:40.393488 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38df9b5c-2de8-4b29-8596-24d7c0a047ad None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:40.413744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38df9b5c-2de8-4b29-8596-24d7c0a047ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-38df9b5c-2de8-4b29-8596-24d7c0a047ad None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:40.418583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:40.419594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:40.423016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 237/442] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:26:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:40.426734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38df9b5c-2de8-4b29-8596-24d7c0a047ad None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:40.492989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e395fed9-a88a-41f5-8db6-5fc228daf1f5 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:40.538876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862a4f6f-f992-4446-88d7-8f961e39db14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:40.539523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862a4f6f-f992-4446-88d7-8f961e39db14 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:40.539801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862a4f6f-f992-4446-88d7-8f961e39db14 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:40.540076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862a4f6f-f992-4446-88d7-8f961e39db14 None admin] PATH_INFO: `/v3/groups/f95b8af0f9ae46c7b420273e832ffb67` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:40.541232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862a4f6f-f992-4446-88d7-8f961e39db14 None admin] RBAC: Authorizing `identity:get_group(group_id=f95b8af0f9ae46c7b420273e832ffb67)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:40.542738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862a4f6f-f992-4446-88d7-8f961e39db14 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:40.544300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 206/443] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:40 2022] GET /identity/v3/groups/f95b8af0f9ae46c7b420273e832ffb67 => generated 226 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:40.545081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862a4f6f-f992-4446-88d7-8f961e39db14 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:40.552042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38df9b5c-2de8-4b29-8596-24d7c0a047ad None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:40.574440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:40.595530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fe9a56-7989-4fa6-a411-e9bcf4251b77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:40.596143 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe9a56-7989-4fa6-a411-e9bcf4251b77 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:40.596417 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe9a56-7989-4fa6-a411-e9bcf4251b77 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:40.596684 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe9a56-7989-4fa6-a411-e9bcf4251b77 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:40.597392 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fe9a56-7989-4fa6-a411-e9bcf4251b77 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:40.597691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fe9a56-7989-4fa6-a411-e9bcf4251b77 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:40.599145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fe9a56-7989-4fa6-a411-e9bcf4251b77 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:40.600185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:40.609362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 238/444] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:40.610022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fe9a56-7989-4fa6-a411-e9bcf4251b77 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:40.668102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862a4f6f-f992-4446-88d7-8f961e39db14 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:40.711217 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92b9300-3ac2-4274-912d-0f93c393ccd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:40.712249 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92b9300-3ac2-4274-912d-0f93c393ccd5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:40.712690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92b9300-3ac2-4274-912d-0f93c393ccd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:40.712966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92b9300-3ac2-4274-912d-0f93c393ccd5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:40.729089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92b9300-3ac2-4274-912d-0f93c393ccd5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:40.730531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fe9a56-7989-4fa6-a411-e9bcf4251b77 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-e92b9300-3ac2-4274-912d-0f93c393ccd5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:40.735178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:40.736078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:40.736856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 207/445] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:26:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:40.738062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92b9300-3ac2-4274-912d-0f93c393ccd5 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:40.766434 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0664da-c9ed-417f-b798-0a5394610162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:40.767067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0664da-c9ed-417f-b798-0a5394610162 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:40.767348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0664da-c9ed-417f-b798-0a5394610162 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:40.767624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0664da-c9ed-417f-b798-0a5394610162 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:40.768351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0664da-c9ed-417f-b798-0a5394610162 None admin] RBAC: Adding query filter params (group.id=f95b8af0f9ae46c7b420273e832ffb67, role.id=9912acfb698e41ebbce6a5147fafe896, scope.project.id=5c50437619304d0d857f65feed1d19e3) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:40.768659 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0664da-c9ed-417f-b798-0a5394610162 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:40.770207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0664da-c9ed-417f-b798-0a5394610162 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:40.776029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 239/446] 10.222.0.65 () {56 vars in 1376 bytes} [Wed Aug 24 03:26:40 2022] GET /identity/v3/role_assignments?group.id=f95b8af0f9ae46c7b420273e832ffb67&scope.project.id=5c50437619304d0d857f65feed1d19e3&role.id=9912acfb698e41ebbce6a5147fafe896 => generated 611 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:40.776896 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0664da-c9ed-417f-b798-0a5394610162 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:40.828837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:40.860569 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92b9300-3ac2-4274-912d-0f93c393ccd5 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:40.864606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:40.892703 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0664da-c9ed-417f-b798-0a5394610162 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:40.897726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3d039e-42b5-4d66-9682-600213f80e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:40.898881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3d039e-42b5-4d66-9682-600213f80e00 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:40.899370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3d039e-42b5-4d66-9682-600213f80e00 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:40.900004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3d039e-42b5-4d66-9682-600213f80e00 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:40.935148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72df525a-bd2c-49ba-a50b-1b66041ec28e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:40.935753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72df525a-bd2c-49ba-a50b-1b66041ec28e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:40.936037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72df525a-bd2c-49ba-a50b-1b66041ec28e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:40.936304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72df525a-bd2c-49ba-a50b-1b66041ec28e None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:40.936960 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72df525a-bd2c-49ba-a50b-1b66041ec28e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:40.937256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72df525a-bd2c-49ba-a50b-1b66041ec28e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:40.938723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72df525a-bd2c-49ba-a50b-1b66041ec28e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:40.949074 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 240/447] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:40.954605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72df525a-bd2c-49ba-a50b-1b66041ec28e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:40.984693 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3d039e-42b5-4d66-9682-600213f80e00 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:40.987093 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3d039e-42b5-4d66-9682-600213f80e00 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:41.074010 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72df525a-bd2c-49ba-a50b-1b66041ec28e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:41.085203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:41.106041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 208/448] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:41.112456 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3d039e-42b5-4d66-9682-600213f80e00 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:41.116173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:41.122872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7c13f0-8580-40dd-930a-ed87bfcc8b65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:41.123559 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7c13f0-8580-40dd-930a-ed87bfcc8b65 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:41.123839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7c13f0-8580-40dd-930a-ed87bfcc8b65 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:41.124136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7c13f0-8580-40dd-930a-ed87bfcc8b65 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:41.195525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7c13f0-8580-40dd-930a-ed87bfcc8b65 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:41.196981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7c13f0-8580-40dd-930a-ed87bfcc8b65 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:41.213504 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3d039e-42b5-4d66-9682-600213f80e00 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:41.247775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d22189-0c38-4dbc-a63b-49036f56edf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:41.248402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d22189-0c38-4dbc-a63b-49036f56edf1 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:41.249008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d22189-0c38-4dbc-a63b-49036f56edf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:41.249008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d22189-0c38-4dbc-a63b-49036f56edf1 None admin] PATH_INFO: `/v3/projects/5c50437619304d0d857f65feed1d19e3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:41.250058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d22189-0c38-4dbc-a63b-49036f56edf1 None admin] RBAC: Authorizing `identity:get_project(project_id=5c50437619304d0d857f65feed1d19e3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:41.251577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d22189-0c38-4dbc-a63b-49036f56edf1 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:41.253281 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 209/449] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:41 2022] GET /identity/v3/projects/5c50437619304d0d857f65feed1d19e3 => generated 302 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:41.259779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d22189-0c38-4dbc-a63b-49036f56edf1 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:41.296740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 241/450] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 24 03:26:41.302560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7c13f0-8580-40dd-930a-ed87bfcc8b65 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:41.310321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:41.329110 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:41.359444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d22189-0c38-4dbc-a63b-49036f56edf1 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:41.393323 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356bdee6-2ff3-4a7e-b91a-ffed7b6f5cc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:41.394563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356bdee6-2ff3-4a7e-b91a-ffed7b6f5cc0 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:41.395117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356bdee6-2ff3-4a7e-b91a-ffed7b6f5cc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:41.395308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356bdee6-2ff3-4a7e-b91a-ffed7b6f5cc0 None admin] PATH_INFO: `/v3/groups/f95b8af0f9ae46c7b420273e832ffb67` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:41.396468 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356bdee6-2ff3-4a7e-b91a-ffed7b6f5cc0 None admin] RBAC: Authorizing `identity:get_group(group_id=f95b8af0f9ae46c7b420273e832ffb67)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:41.397840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356bdee6-2ff3-4a7e-b91a-ffed7b6f5cc0 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:41.399559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 210/451] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:41 2022] GET /identity/v3/groups/f95b8af0f9ae46c7b420273e832ffb67 => generated 226 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:41.400109 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7c13f0-8580-40dd-930a-ed87bfcc8b65 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:41.405117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356bdee6-2ff3-4a7e-b91a-ffed7b6f5cc0 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:41.437773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830b20cf-83ec-47ba-bb99-50bedc9fdf78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:41.438433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830b20cf-83ec-47ba-bb99-50bedc9fdf78 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:41.438710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830b20cf-83ec-47ba-bb99-50bedc9fdf78 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:41.438983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830b20cf-83ec-47ba-bb99-50bedc9fdf78 None admin] PATH_INFO: `/v3/projects/b8d28080f87c4783b9ef971b31a0d51a` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:41.440107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830b20cf-83ec-47ba-bb99-50bedc9fdf78 None admin] RBAC: Authorizing `identity:get_project(project_id=b8d28080f87c4783b9ef971b31a0d51a)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:41.441677 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830b20cf-83ec-47ba-bb99-50bedc9fdf78 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:41.443507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 242/452] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:26:41 2022] GET /identity/v3/projects/b8d28080f87c4783b9ef971b31a0d51a => generated 344 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:41.448917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830b20cf-83ec-47ba-bb99-50bedc9fdf78 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:41.507545 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356bdee6-2ff3-4a7e-b91a-ffed7b6f5cc0 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:41.523206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:41.542389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a1c38-d404-43d2-b5c2-d0243968f13e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:41.542994 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a1c38-d404-43d2-b5c2-d0243968f13e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:41.543299 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a1c38-d404-43d2-b5c2-d0243968f13e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:41.543578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a1c38-d404-43d2-b5c2-d0243968f13e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:41.544300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6a1c38-d404-43d2-b5c2-d0243968f13e None admin] RBAC: Adding query filter params (group.id=f95b8af0f9ae46c7b420273e832ffb67, role.id=d02a8e6c49b74aaa9403f8bd535b90bf, scope.project.id=5c50437619304d0d857f65feed1d19e3) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:41.544608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6a1c38-d404-43d2-b5c2-d0243968f13e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:41.546037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6a1c38-d404-43d2-b5c2-d0243968f13e None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:41.547752 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830b20cf-83ec-47ba-bb99-50bedc9fdf78 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:41.552743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 211/453] 10.222.0.65 () {56 vars in 1376 bytes} [Wed Aug 24 03:26:41 2022] GET /identity/v3/role_assignments?group.id=f95b8af0f9ae46c7b420273e832ffb67&scope.project.id=5c50437619304d0d857f65feed1d19e3&role.id=d02a8e6c49b74aaa9403f8bd535b90bf => generated 611 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:41.584586 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655ff903-4c12-4e74-88c0-ee506a5d3888 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:41.585242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655ff903-4c12-4e74-88c0-ee506a5d3888 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:41.585519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655ff903-4c12-4e74-88c0-ee506a5d3888 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:41.585789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655ff903-4c12-4e74-88c0-ee506a5d3888 None admin] PATH_INFO: `/v3/groups/2750e7b66ccf40bbb43b081af3edde7b` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:41.587025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655ff903-4c12-4e74-88c0-ee506a5d3888 None admin] RBAC: Authorizing `identity:get_group(group_id=2750e7b66ccf40bbb43b081af3edde7b)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:41.588520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655ff903-4c12-4e74-88c0-ee506a5d3888 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:41.590346 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 243/454] 10.222.0.65 () {56 vars in 1180 bytes} [Wed Aug 24 03:26:41 2022] GET /identity/v3/groups/2750e7b66ccf40bbb43b081af3edde7b => generated 229 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:41.596142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a1c38-d404-43d2-b5c2-d0243968f13e None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:41.600246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:41.701464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a1c38-d404-43d2-b5c2-d0243968f13e None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:41.735247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bb22ea-0bcf-41df-913d-741e9456514d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:41.735855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bb22ea-0bcf-41df-913d-741e9456514d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:41.736143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bb22ea-0bcf-41df-913d-741e9456514d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:41.736414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bb22ea-0bcf-41df-913d-741e9456514d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:41.737143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bb22ea-0bcf-41df-913d-741e9456514d None admin] RBAC: Adding query filter params (group.id=2750e7b66ccf40bbb43b081af3edde7b, role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=b8d28080f87c4783b9ef971b31a0d51a) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:41.737469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bb22ea-0bcf-41df-913d-741e9456514d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:41.739007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bb22ea-0bcf-41df-913d-741e9456514d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:41.744939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 212/455] 10.222.0.65 () {56 vars in 1376 bytes} [Wed Aug 24 03:26:41 2022] GET /identity/v3/role_assignments?group.id=2750e7b66ccf40bbb43b081af3edde7b&scope.project.id=b8d28080f87c4783b9ef971b31a0d51a&role.id=43ca25467bc6484da3a50508efb79230 => generated 611 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:44.161356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb2ef38-2ef6-4a6c-a9d9-fc702edb819a None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:44.161875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb2ef38-2ef6-4a6c-a9d9-fc702edb819a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:44.161875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb2ef38-2ef6-4a6c-a9d9-fc702edb819a None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:44.163523 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 244/456] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:26:44.176272 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32ad878-01d1-4947-b562-5e3796ba9289 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:44.176800 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32ad878-01d1-4947-b562-5e3796ba9289 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:44.177013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32ad878-01d1-4947-b562-5e3796ba9289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:44.198136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-c32ad878-01d1-4947-b562-5e3796ba9289 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:44.199386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c32ad878-01d1-4947-b562-5e3796ba9289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:44.232102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf14509-8101-4966-a5ae-3ba7bea18677 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:44.232424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf14509-8101-4966-a5ae-3ba7bea18677 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:44.232653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf14509-8101-4966-a5ae-3ba7bea18677 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:44.234277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 245/457] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:44.240094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69692b4b-d967-4ad8-823a-0fbdf001b711 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:44.240386 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69692b4b-d967-4ad8-823a-0fbdf001b711 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:44.248591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69692b4b-d967-4ad8-823a-0fbdf001b711 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:44.271736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-69692b4b-d967-4ad8-823a-0fbdf001b711 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:44.272940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-69692b4b-d967-4ad8-823a-0fbdf001b711 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:44.301818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:44.314859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 213/458] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:44.372714 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 246/459] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:44.500801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da3da2-42e3-4bb8-b701-113215079acb None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:44.501422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da3da2-42e3-4bb8-b701-113215079acb None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:44.501884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5da3da2-42e3-4bb8-b701-113215079acb None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:44.503497 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 214/460] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:44.509265 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2798c4-6d13-427a-8ced-cb0fd239f910 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:44.509557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2798c4-6d13-427a-8ced-cb0fd239f910 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:44.509830 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2798c4-6d13-427a-8ced-cb0fd239f910 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:44.535360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-ce2798c4-6d13-427a-8ced-cb0fd239f910 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:44.536553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ce2798c4-6d13-427a-8ced-cb0fd239f910 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:44.620749 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 247/461] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:44.679120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5da3da2-42e3-4bb8-b701-113215079acb None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:44.708496 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2798c4-6d13-427a-8ced-cb0fd239f910 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:44.711647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:44.853032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5da3da2-42e3-4bb8-b701-113215079acb None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:44.881136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2798c4-6d13-427a-8ced-cb0fd239f910 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:44.888891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd2ac20-7946-435d-bc7c-3b45b4dd341d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:44.889787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd2ac20-7946-435d-bc7c-3b45b4dd341d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:44.890297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd2ac20-7946-435d-bc7c-3b45b4dd341d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:44.890763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd2ac20-7946-435d-bc7c-3b45b4dd341d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:44.901815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd2ac20-7946-435d-bc7c-3b45b4dd341d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:44.903533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd2ac20-7946-435d-bc7c-3b45b4dd341d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:44.917461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7498ea0f-324c-4b89-99c7-23e3d6004b86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:44.918335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7498ea0f-324c-4b89-99c7-23e3d6004b86 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:44.918757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7498ea0f-324c-4b89-99c7-23e3d6004b86 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:44.919185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7498ea0f-324c-4b89-99c7-23e3d6004b86 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-9cd2ac20-7946-435d-bc7c-3b45b4dd341d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:44.919542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:44.920405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:44.921254 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:44.921254 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:44.921254 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:44.923234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 215/462] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:44.924578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b907b125-d6de-4a43-9715-c337e0a04379 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:44.925105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b907b125-d6de-4a43-9715-c337e0a04379 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:44.925660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b907b125-d6de-4a43-9715-c337e0a04379 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:44.927335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 216/463] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:44.932718 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df77ed49-b98a-4d57-8d96-014c8b31e0c4 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:44.933667 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df77ed49-b98a-4d57-8d96-014c8b31e0c4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:44.934099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df77ed49-b98a-4d57-8d96-014c8b31e0c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:44.935298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7498ea0f-324c-4b89-99c7-23e3d6004b86 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:44.936843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7498ea0f-324c-4b89-99c7-23e3d6004b86 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-7498ea0f-324c-4b89-99c7-23e3d6004b86 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:44.954949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:44.955893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:44.957410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:44.957410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:44.957410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:44.957586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-df77ed49-b98a-4d57-8d96-014c8b31e0c4 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:44.957656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 248/464] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:44.957719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-df77ed49-b98a-4d57-8d96-014c8b31e0c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:44.958076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7498ea0f-324c-4b89-99c7-23e3d6004b86 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:45.007874 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:45.019763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:45.028097 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 217/465] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:26:45.029072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df77ed49-b98a-4d57-8d96-014c8b31e0c4 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:45.068014 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7498ea0f-324c-4b89-99c7-23e3d6004b86 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:45.109436 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f40cbf-a315-4802-ac06-1da3d4fddc6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:45.110642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f40cbf-a315-4802-ac06-1da3d4fddc6b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:45.111545 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f40cbf-a315-4802-ac06-1da3d4fddc6b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:45.112257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f40cbf-a315-4802-ac06-1da3d4fddc6b None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:45.113476 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f40cbf-a315-4802-ac06-1da3d4fddc6b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:45.113956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f40cbf-a315-4802-ac06-1da3d4fddc6b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:45.116200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f40cbf-a315-4802-ac06-1da3d4fddc6b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:45.132655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 249/466] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:45.133741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f40cbf-a315-4802-ac06-1da3d4fddc6b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:45.164654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df77ed49-b98a-4d57-8d96-014c8b31e0c4 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:45.202408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732a2223-5869-43e4-b01a-d3df66b24779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:45.203024 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732a2223-5869-43e4-b01a-d3df66b24779 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:45.203383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732a2223-5869-43e4-b01a-d3df66b24779 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:45.203728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732a2223-5869-43e4-b01a-d3df66b24779 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:45.215278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732a2223-5869-43e4-b01a-d3df66b24779 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:45.216862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732a2223-5869-43e4-b01a-d3df66b24779 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-732a2223-5869-43e4-b01a-d3df66b24779 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:45.233475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:45.234411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:45.235262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:45.235262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:45.235262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:45.235262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 218/467] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:45.236220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732a2223-5869-43e4-b01a-d3df66b24779 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:45.271324 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:45.277049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f40cbf-a315-4802-ac06-1da3d4fddc6b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:45.296498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:45.324563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ee5e8c-4cc1-4232-aac8-74c088854d8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:45.325677 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ee5e8c-4cc1-4232-aac8-74c088854d8a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:45.326620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ee5e8c-4cc1-4232-aac8-74c088854d8a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:45.327360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ee5e8c-4cc1-4232-aac8-74c088854d8a None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:45.328182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ee5e8c-4cc1-4232-aac8-74c088854d8a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:45.328844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ee5e8c-4cc1-4232-aac8-74c088854d8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:45.330810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ee5e8c-4cc1-4232-aac8-74c088854d8a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:45.344754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732a2223-5869-43e4-b01a-d3df66b24779 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:45.345817 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 250/468] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:45.348355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ee5e8c-4cc1-4232-aac8-74c088854d8a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:45.384258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df07c2a8-f1af-431b-88ab-a93899e19596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:45.384862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df07c2a8-f1af-431b-88ab-a93899e19596 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:45.385139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df07c2a8-f1af-431b-88ab-a93899e19596 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:45.385442 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df07c2a8-f1af-431b-88ab-a93899e19596 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:45.386045 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df07c2a8-f1af-431b-88ab-a93899e19596 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:45.387507 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df07c2a8-f1af-431b-88ab-a93899e19596 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:45.457335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ee5e8c-4cc1-4232-aac8-74c088854d8a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:45.490560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643e7663-0a5c-4282-8e46-9948c7edf750 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:45.491169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643e7663-0a5c-4282-8e46-9948c7edf750 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:45.491450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643e7663-0a5c-4282-8e46-9948c7edf750 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:45.491724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643e7663-0a5c-4282-8e46-9948c7edf750 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:45.492365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643e7663-0a5c-4282-8e46-9948c7edf750 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:45.492675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643e7663-0a5c-4282-8e46-9948c7edf750 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:45.494076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643e7663-0a5c-4282-8e46-9948c7edf750 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:45.514183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 219/469] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:26:45 2022] POST /identity/v3/users => generated 246 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:26:45.514393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 251/470] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:45.514483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df07c2a8-f1af-431b-88ab-a93899e19596 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:45.514591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643e7663-0a5c-4282-8e46-9948c7edf750 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:45.556484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:45.632929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643e7663-0a5c-4282-8e46-9948c7edf750 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:45.645179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:45.663543 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df07c2a8-f1af-431b-88ab-a93899e19596 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:45.672447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e980e56b-3062-4058-b399-9169f350078f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:45.673350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e980e56b-3062-4058-b399-9169f350078f None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:45.673770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e980e56b-3062-4058-b399-9169f350078f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:45.674041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e980e56b-3062-4058-b399-9169f350078f None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:45.674739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e980e56b-3062-4058-b399-9169f350078f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:45.676934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e980e56b-3062-4058-b399-9169f350078f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:45.704897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e57947-53c3-4b19-9352-1984d4a6db0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:45.705517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e57947-53c3-4b19-9352-1984d4a6db0a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:45.705808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e57947-53c3-4b19-9352-1984d4a6db0a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:45.706055 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e57947-53c3-4b19-9352-1984d4a6db0a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:45.716551 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e57947-53c3-4b19-9352-1984d4a6db0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:45.717924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e57947-53c3-4b19-9352-1984d4a6db0a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-12e57947-53c3-4b19-9352-1984d4a6db0a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:45.734670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:45.735807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:45.736858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:45.736858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:45.736858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:45.736858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 220/471] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:45.736858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e57947-53c3-4b19-9352-1984d4a6db0a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:45.785804 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 252/472] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:26:45 2022] POST /identity/v3/users => generated 248 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:26:45.786980 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecba4fd-2856-4a5f-a037-d58aa124f315 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:45.787263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecba4fd-2856-4a5f-a037-d58aa124f315 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:45.787532 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecba4fd-2856-4a5f-a037-d58aa124f315 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:45.793157 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 253/473] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:45 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:45.794227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecba4fd-2856-4a5f-a037-d58aa124f315 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:45.854012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e57947-53c3-4b19-9352-1984d4a6db0a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:45.895408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4777b0ad-b4e2-4706-bb5f-60a8c9ad94cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:45.896415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4777b0ad-b4e2-4706-bb5f-60a8c9ad94cc None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:45.896883 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4777b0ad-b4e2-4706-bb5f-60a8c9ad94cc None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:45.897444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4777b0ad-b4e2-4706-bb5f-60a8c9ad94cc None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:45.898432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4777b0ad-b4e2-4706-bb5f-60a8c9ad94cc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:45.899991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4777b0ad-b4e2-4706-bb5f-60a8c9ad94cc None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:45.913534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:45.927482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:45.938522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecba4fd-2856-4a5f-a037-d58aa124f315 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:45.976184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aad00f2-e79f-40dc-972b-6b6718fa03bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:45.976184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aad00f2-e79f-40dc-972b-6b6718fa03bd None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:45.976184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aad00f2-e79f-40dc-972b-6b6718fa03bd None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:45.976501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aad00f2-e79f-40dc-972b-6b6718fa03bd None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:45.978562 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aad00f2-e79f-40dc-972b-6b6718fa03bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:45.978562 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aad00f2-e79f-40dc-972b-6b6718fa03bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:45.985642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aad00f2-e79f-40dc-972b-6b6718fa03bd None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:45.991441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 254/474] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:45.992651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ae53e-5fec-4b3d-9cf6-bb0386d1a688 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:45.992995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ae53e-5fec-4b3d-9cf6-bb0386d1a688 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:45.993272 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ae53e-5fec-4b3d-9cf6-bb0386d1a688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:46.011983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-143ae53e-5fec-4b3d-9cf6-bb0386d1a688 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:46.013191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-143ae53e-5fec-4b3d-9cf6-bb0386d1a688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:46.016814 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 221/475] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:26:45 2022] POST /identity/v3/users => generated 248 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:26:46.017732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4777b0ad-b4e2-4706-bb5f-60a8c9ad94cc None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:46.103692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 255/476] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:46.157933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4777b0ad-b4e2-4706-bb5f-60a8c9ad94cc None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:46.196974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2eddc0-596c-4285-9520-f9e2a3af80c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:46.197613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2eddc0-596c-4285-9520-f9e2a3af80c4 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:46.197906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2eddc0-596c-4285-9520-f9e2a3af80c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:46.198183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2eddc0-596c-4285-9520-f9e2a3af80c4 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:46.198901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2eddc0-596c-4285-9520-f9e2a3af80c4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:46.200415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2eddc0-596c-4285-9520-f9e2a3af80c4 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:46.254484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 222/477] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:26:46 2022] POST /identity/v3/users => generated 249 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:26:46.632325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143ae53e-5fec-4b3d-9cf6-bb0386d1a688 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:46.720309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:46.742115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143ae53e-5fec-4b3d-9cf6-bb0386d1a688 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:46.783211 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ec2741-bbdf-4966-8ad0-6d4363c2b4a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:46.783821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ec2741-bbdf-4966-8ad0-6d4363c2b4a3 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:46.784099 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ec2741-bbdf-4966-8ad0-6d4363c2b4a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:46.784452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ec2741-bbdf-4966-8ad0-6d4363c2b4a3 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:46.785125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ec2741-bbdf-4966-8ad0-6d4363c2b4a3 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:46.786636 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ec2741-bbdf-4966-8ad0-6d4363c2b4a3 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:46.818860 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.sql.core [None req-f5ec2741-bbdf-4966-8ad0-6d4363c2b4a3 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 24 03:26:46.818860 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 24 03:26:46.818860 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [parameters: {'id': '21b35a53b3964444afaf435336d46c3a', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 24 03:26:46.818860 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81657) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-f5ec2741-bbdf-4966-8ad0-6d4363c2b4a3 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application result = self._query(query) Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application conn.query(q) Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application result.read() Aug 24 03:26:46.843631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return ref.to_dict() Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 24 03:26:46.844546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application next(self.gen) Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application yield resource Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application next(self.gen) Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self.session.rollback() Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self.force_reraise() Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise self.value Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application session.commit() Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 24 03:26:46.845455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._prepare_impl() Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self.session.flush() Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._flush(objects) Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application compat.raise_( Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application flush_context.execute() Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rec.execute(self) Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 24 03:26:46.846334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 24 03:26:46.847774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application result = self._query(query) Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application conn.query(q) Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application result.read() Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 24 03:26:46.848712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [parameters: {'id': '21b35a53b3964444afaf435336d46c3a', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:46.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 24 03:26:46.855878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:46.858087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 256/478] 10.222.0.65 () {60 vars in 1131 bytes} [Wed Aug 24 03:26:46 2022] POST /identity/v3/roles => generated 144 bytes in 213 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:46.858248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2eddc0-596c-4285-9520-f9e2a3af80c4 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:46.943887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:47.004358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2eddc0-596c-4285-9520-f9e2a3af80c4 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:47.039989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68950ae9-2ba7-4672-b0a8-ffca8ca03e0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:47.040633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68950ae9-2ba7-4672-b0a8-ffca8ca03e0c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:47.040910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68950ae9-2ba7-4672-b0a8-ffca8ca03e0c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:47.041179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68950ae9-2ba7-4672-b0a8-ffca8ca03e0c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:47.052204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68950ae9-2ba7-4672-b0a8-ffca8ca03e0c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-68950ae9-2ba7-4672-b0a8-ffca8ca03e0c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:47.057344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:47.058261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:47.059089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 223/479] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:26:46 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:47.064376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ec2741-bbdf-4966-8ad0-6d4363c2b4a3 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:47.223697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ec2741-bbdf-4966-8ad0-6d4363c2b4a3 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:47.265775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14b48a1-c23c-4d50-9393-20b6200e8ff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:47.266415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14b48a1-c23c-4d50-9393-20b6200e8ff5 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:47.266694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14b48a1-c23c-4d50-9393-20b6200e8ff5 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:47.266965 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14b48a1-c23c-4d50-9393-20b6200e8ff5 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:47.267663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14b48a1-c23c-4d50-9393-20b6200e8ff5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:47.267966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14b48a1-c23c-4d50-9393-20b6200e8ff5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:47.270095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14b48a1-c23c-4d50-9393-20b6200e8ff5 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:47.279238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 257/480] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:47 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:47.886429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31834740-6069-40da-ab3a-0ecaa52fb600 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:47.886942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31834740-6069-40da-ab3a-0ecaa52fb600 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:47.886942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31834740-6069-40da-ab3a-0ecaa52fb600 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:47.888578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 224/481] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:47.893944 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c1fd50-5f06-4fef-8fe4-363721cb097a None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:47.894589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c1fd50-5f06-4fef-8fe4-363721cb097a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:47.894589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c1fd50-5f06-4fef-8fe4-363721cb097a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:47.911292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:47.919143 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-11c1fd50-5f06-4fef-8fe4-363721cb097a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:47.920352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-11c1fd50-5f06-4fef-8fe4-363721cb097a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:48.009717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 258/482] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:48.232171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28461b90-d625-4f17-b28f-ca94f37dc092 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:48.232458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28461b90-d625-4f17-b28f-ca94f37dc092 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:48.232633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28461b90-d625-4f17-b28f-ca94f37dc092 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:48.234282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 225/483] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:48.240073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178fbfff-e16a-4a08-9419-74238ae5a6c2 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:48.246376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178fbfff-e16a-4a08-9419-74238ae5a6c2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:48.246376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178fbfff-e16a-4a08-9419-74238ae5a6c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:48.263598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-178fbfff-e16a-4a08-9419-74238ae5a6c2 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:48.264786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-178fbfff-e16a-4a08-9419-74238ae5a6c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:48.342379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28461b90-d625-4f17-b28f-ca94f37dc092 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:48.344561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 259/484] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:48.451673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28461b90-d625-4f17-b28f-ca94f37dc092 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:48.466828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:48.486785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c930716-afa6-4200-86e9-87f4d772fd0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:48.487371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c930716-afa6-4200-86e9-87f4d772fd0f None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:48.487752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c930716-afa6-4200-86e9-87f4d772fd0f None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:48.488033 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c930716-afa6-4200-86e9-87f4d772fd0f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:48.498938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c930716-afa6-4200-86e9-87f4d772fd0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-4c930716-afa6-4200-86e9-87f4d772fd0f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:48.504321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:48.505392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:48.506666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 226/485] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:26:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:48.511390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178fbfff-e16a-4a08-9419-74238ae5a6c2 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:48.606783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:48.652316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178fbfff-e16a-4a08-9419-74238ae5a6c2 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:48.701216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e750823c-b3a6-438b-b15f-498d02210684 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:48.701824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e750823c-b3a6-438b-b15f-498d02210684 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:48.702104 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e750823c-b3a6-438b-b15f-498d02210684 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:48.702397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e750823c-b3a6-438b-b15f-498d02210684 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:48.703067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e750823c-b3a6-438b-b15f-498d02210684 None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:48.703363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e750823c-b3a6-438b-b15f-498d02210684 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:48.704741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e750823c-b3a6-438b-b15f-498d02210684 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:48.714438 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c930716-afa6-4200-86e9-87f4d772fd0f None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:48.715237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 260/486] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:26:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:26:48.720215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e750823c-b3a6-438b-b15f-498d02210684 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:48.908869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e750823c-b3a6-438b-b15f-498d02210684 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:48.951207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:48.951362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c930716-afa6-4200-86e9-87f4d772fd0f None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:48.963346 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7836cba-906d-4de8-a4f3-7828591ce9f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:48.964364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7836cba-906d-4de8-a4f3-7828591ce9f9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:48.965015 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7836cba-906d-4de8-a4f3-7828591ce9f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:48.969200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7836cba-906d-4de8-a4f3-7828591ce9f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:48.978548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7836cba-906d-4de8-a4f3-7828591ce9f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:48.980003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7836cba-906d-4de8-a4f3-7828591ce9f9 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:48.994715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27539e4-6841-4477-aa91-340397a55b43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:48.995344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27539e4-6841-4477-aa91-340397a55b43 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:48.995617 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27539e4-6841-4477-aa91-340397a55b43 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:48.995881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27539e4-6841-4477-aa91-340397a55b43 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-c7836cba-906d-4de8-a4f3-7828591ce9f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:48.996429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:48.997365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:48.998236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:48.998236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:48.998236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:48.998236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 261/487] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:48.999447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b6df03-39ea-4612-8744-7798f4cad9e6 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:48.999731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b6df03-39ea-4612-8744-7798f4cad9e6 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.000002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b6df03-39ea-4612-8744-7798f4cad9e6 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.006334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 262/488] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:49.006334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdb6655-1d05-49c8-beab-8368c039fa31 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:49.006334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdb6655-1d05-49c8-beab-8368c039fa31 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.006334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdb6655-1d05-49c8-beab-8368c039fa31 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.006334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 263/489] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:49.009942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdb6655-1d05-49c8-beab-8368c039fa31 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:49.012588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27539e4-6841-4477-aa91-340397a55b43 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-b27539e4-6841-4477-aa91-340397a55b43 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:49.017738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:49.019028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:49.020267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 227/490] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:26:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:49.020929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be66abf-cb4b-4204-b8fc-f57e271a2f85 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:49.020929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be66abf-cb4b-4204-b8fc-f57e271a2f85 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.020929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be66abf-cb4b-4204-b8fc-f57e271a2f85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.040904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-5be66abf-cb4b-4204-b8fc-f57e271a2f85 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:49.042619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5be66abf-cb4b-4204-b8fc-f57e271a2f85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:49.073627 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:49.119680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 228/491] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:49.121370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f3ec2-cdfe-4b2e-bc8f-7bc5ffffd7c9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:49.121666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f3ec2-cdfe-4b2e-bc8f-7bc5ffffd7c9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.121939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f3ec2-cdfe-4b2e-bc8f-7bc5ffffd7c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.135150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdb6655-1d05-49c8-beab-8368c039fa31 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:49.141935 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-961f3ec2-cdfe-4b2e-bc8f-7bc5ffffd7c9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:49.143510 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-961f3ec2-cdfe-4b2e-bc8f-7bc5ffffd7c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:49.180940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0aa4bb-51e3-4fb8-a1b6-92a0e5f286d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:49.181599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0aa4bb-51e3-4fb8-a1b6-92a0e5f286d9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:49.181973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0aa4bb-51e3-4fb8-a1b6-92a0e5f286d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.182363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0aa4bb-51e3-4fb8-a1b6-92a0e5f286d9 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.183349 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0aa4bb-51e3-4fb8-a1b6-92a0e5f286d9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:49.184584 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0aa4bb-51e3-4fb8-a1b6-92a0e5f286d9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:49.186052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0aa4bb-51e3-4fb8-a1b6-92a0e5f286d9 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:49.206676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 264/492] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:49.208090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0aa4bb-51e3-4fb8-a1b6-92a0e5f286d9 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:49.222998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 229/493] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:49.224202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961f3ec2-cdfe-4b2e-bc8f-7bc5ffffd7c9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:49.283021 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:49.372930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:49.383216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961f3ec2-cdfe-4b2e-bc8f-7bc5ffffd7c9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:49.400075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0aa4bb-51e3-4fb8-a1b6-92a0e5f286d9 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:49.432592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0650bc8-dbe0-4f23-8039-94db51141cba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:49.433423 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0650bc8-dbe0-4f23-8039-94db51141cba None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:49.433851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0650bc8-dbe0-4f23-8039-94db51141cba None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.434388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0650bc8-dbe0-4f23-8039-94db51141cba None admin] PATH_INFO: `/v3/users/nova` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.442426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0650bc8-dbe0-4f23-8039-94db51141cba None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:49.444260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0650bc8-dbe0-4f23-8039-94db51141cba None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-e0650bc8-dbe0-4f23-8039-94db51141cba None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:26:49.458748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:49.459685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:49.460575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:49.460575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:49.460575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:26:49.460575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:26:49.460575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:26:49.460575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:26:49.460575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:26:49.460575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:26:49.460575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:26:49.460575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:49.462854 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 230/494] 10.222.0.65 () {56 vars in 1093 bytes} [Wed Aug 24 03:26:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:49.468497 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0650bc8-dbe0-4f23-8039-94db51141cba None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:49.486202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3367ce-960a-4f9a-bcf7-248c4aabe88f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:49.486775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3367ce-960a-4f9a-bcf7-248c4aabe88f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:49.487056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3367ce-960a-4f9a-bcf7-248c4aabe88f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.487925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3367ce-960a-4f9a-bcf7-248c4aabe88f None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.488820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3367ce-960a-4f9a-bcf7-248c4aabe88f None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:49.489121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3367ce-960a-4f9a-bcf7-248c4aabe88f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:49.490549 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3367ce-960a-4f9a-bcf7-248c4aabe88f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:49.503578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 265/495] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:26:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:49.504528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3367ce-960a-4f9a-bcf7-248c4aabe88f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:49.575994 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0650bc8-dbe0-4f23-8039-94db51141cba None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:49.611262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3367ce-960a-4f9a-bcf7-248c4aabe88f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:49.611722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963f1c48-f7af-45ab-9120-609fe67698c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:49.612364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963f1c48-f7af-45ab-9120-609fe67698c0 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:49.613148 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963f1c48-f7af-45ab-9120-609fe67698c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.613355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:49.613450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963f1c48-f7af-45ab-9120-609fe67698c0 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.614346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963f1c48-f7af-45ab-9120-609fe67698c0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:49.614653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963f1c48-f7af-45ab-9120-609fe67698c0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:49.616200 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963f1c48-f7af-45ab-9120-609fe67698c0 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:49.649845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c307255-84a7-45b0-8ae1-17f2a4fbc7a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:49.650507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c307255-84a7-45b0-8ae1-17f2a4fbc7a4 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:49.651303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c307255-84a7-45b0-8ae1-17f2a4fbc7a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.651620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c307255-84a7-45b0-8ae1-17f2a4fbc7a4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.659095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 231/496] 10.222.0.65 () {56 vars in 1133 bytes} [Wed Aug 24 03:26:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:49.659095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963f1c48-f7af-45ab-9120-609fe67698c0 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:49.666851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c307255-84a7-45b0-8ae1-17f2a4fbc7a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:49.671669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-3c307255-84a7-45b0-8ae1-17f2a4fbc7a4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:49.672029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:49.672926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:49.673706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 266/497] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:26:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:49.674418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c307255-84a7-45b0-8ae1-17f2a4fbc7a4 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:49.769891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963f1c48-f7af-45ab-9120-609fe67698c0 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:49.782251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c307255-84a7-45b0-8ae1-17f2a4fbc7a4 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:49.806255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96155f2a-e3fd-4d11-90d8-b71fb89b28ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:49.806892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96155f2a-e3fd-4d11-90d8-b71fb89b28ff None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:49.807170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96155f2a-e3fd-4d11-90d8-b71fb89b28ff None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.807447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96155f2a-e3fd-4d11-90d8-b71fb89b28ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.818484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9bab27-879e-4bd8-baa1-ed97e6aea6f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:49.818823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96155f2a-e3fd-4d11-90d8-b71fb89b28ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:49.819518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9bab27-879e-4bd8-baa1-ed97e6aea6f2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:49.819993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9bab27-879e-4bd8-baa1-ed97e6aea6f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.820289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96155f2a-e3fd-4d11-90d8-b71fb89b28ff None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:49.820623 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9bab27-879e-4bd8-baa1-ed97e6aea6f2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.831253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9bab27-879e-4bd8-baa1-ed97e6aea6f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-96155f2a-e3fd-4d11-90d8-b71fb89b28ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:49.834922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:49.835993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:49.837138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:49.837138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:49.837138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-9f9bab27-879e-4bd8-baa1-ed97e6aea6f2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:49.839367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:49.840500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:49.841782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:49.841782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:49.841782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:49.841782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:49.841782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:49.841782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:49.841782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:49.841782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:49.841782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:49.841782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:49.841782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:49.841782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:49.842524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 232/498] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:49.842524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96155f2a-e3fd-4d11-90d8-b71fb89b28ff None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:49.844212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 267/499] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:26:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:49.845103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9bab27-879e-4bd8-baa1-ed97e6aea6f2 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:49.872033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:49.912659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:49.953168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96155f2a-e3fd-4d11-90d8-b71fb89b28ff None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:49.953806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9bab27-879e-4bd8-baa1-ed97e6aea6f2 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:49.986481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405abefb-4ec7-4bc2-b23d-7231629d59da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:49.986664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3810d9e3-d470-4bfa-8f16-e3eec10899eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:49.987084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405abefb-4ec7-4bc2-b23d-7231629d59da None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:49.987226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3810d9e3-d470-4bfa-8f16-e3eec10899eb None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:49.987982 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405abefb-4ec7-4bc2-b23d-7231629d59da None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.988088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3810d9e3-d470-4bfa-8f16-e3eec10899eb None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:49.988361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3810d9e3-d470-4bfa-8f16-e3eec10899eb None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.988447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405abefb-4ec7-4bc2-b23d-7231629d59da None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:49.989027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3810d9e3-d470-4bfa-8f16-e3eec10899eb None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:49.989327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3810d9e3-d470-4bfa-8f16-e3eec10899eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:49.990743 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3810d9e3-d470-4bfa-8f16-e3eec10899eb None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:49.998634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 268/500] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:26:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:49.999435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3810d9e3-d470-4bfa-8f16-e3eec10899eb None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:50.055953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405abefb-4ec7-4bc2-b23d-7231629d59da None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:50.057355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405abefb-4ec7-4bc2-b23d-7231629d59da None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:50.081836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:50.097388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3810d9e3-d470-4bfa-8f16-e3eec10899eb None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:50.115921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:50.131904 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a16ea9-c4f0-4eba-8b01-d928715f0815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:50.132710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a16ea9-c4f0-4eba-8b01-d928715f0815 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:50.132817 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a16ea9-c4f0-4eba-8b01-d928715f0815 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:50.133048 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a16ea9-c4f0-4eba-8b01-d928715f0815 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:50.133685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a16ea9-c4f0-4eba-8b01-d928715f0815 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:50.134085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a16ea9-c4f0-4eba-8b01-d928715f0815 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:50.135383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a16ea9-c4f0-4eba-8b01-d928715f0815 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:50.147369 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 269/501] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:50.148269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a16ea9-c4f0-4eba-8b01-d928715f0815 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:50.157307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 233/502] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:50.158577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfb0b63-4942-4da6-b5cf-6aa4ee4167a3 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:50.158862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfb0b63-4942-4da6-b5cf-6aa4ee4167a3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:50.159137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfb0b63-4942-4da6-b5cf-6aa4ee4167a3 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:50.160643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 234/503] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:50.161359 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfb0b63-4942-4da6-b5cf-6aa4ee4167a3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:50.250639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a16ea9-c4f0-4eba-8b01-d928715f0815 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:50.262277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfb0b63-4942-4da6-b5cf-6aa4ee4167a3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:50.285794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daf6b29-975a-4550-8ed9-59bc282d734f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:50.286445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daf6b29-975a-4550-8ed9-59bc282d734f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:50.286721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daf6b29-975a-4550-8ed9-59bc282d734f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:50.286991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daf6b29-975a-4550-8ed9-59bc282d734f None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:50.287646 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daf6b29-975a-4550-8ed9-59bc282d734f None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:50.287945 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daf6b29-975a-4550-8ed9-59bc282d734f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:50.289327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daf6b29-975a-4550-8ed9-59bc282d734f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:50.295375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e050a76-49f9-4cca-8517-b8b9f94b408b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:50.295987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e050a76-49f9-4cca-8517-b8b9f94b408b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:50.296261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e050a76-49f9-4cca-8517-b8b9f94b408b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:50.296527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e050a76-49f9-4cca-8517-b8b9f94b408b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:50.297027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 270/504] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:26:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:50.297794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daf6b29-975a-4550-8ed9-59bc282d734f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:50.311071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:50.317022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e050a76-49f9-4cca-8517-b8b9f94b408b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:50.319486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e050a76-49f9-4cca-8517-b8b9f94b408b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-6e050a76-49f9-4cca-8517-b8b9f94b408b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:50.335652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:50.336697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:50.337870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:50.337870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:50.337870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:50.337870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 235/505] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:50.337870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e050a76-49f9-4cca-8517-b8b9f94b408b None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:50.355170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:50.413239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daf6b29-975a-4550-8ed9-59bc282d734f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:50.442363 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e050a76-49f9-4cca-8517-b8b9f94b408b None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:50.448246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc56ebd-02fb-4bbb-a481-52d783d9434f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:50.448869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc56ebd-02fb-4bbb-a481-52d783d9434f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:50.449144 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc56ebd-02fb-4bbb-a481-52d783d9434f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:50.449413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc56ebd-02fb-4bbb-a481-52d783d9434f None admin] PATH_INFO: `/v3/users/glance` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:50.456721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc56ebd-02fb-4bbb-a481-52d783d9434f None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:50.458160 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc56ebd-02fb-4bbb-a481-52d783d9434f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-fdc56ebd-02fb-4bbb-a481-52d783d9434f None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:26:50.470191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:50.471087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:50.472033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:50.472033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:50.472033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:26:50.472033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:26:50.472033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:26:50.472033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:26:50.472033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:26:50.472033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:26:50.472033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 24 03:26:50.472033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:50.472033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 271/506] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:50 2022] GET /identity/v3/users/glance => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:50.472642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625ed06b-53c9-419e-a554-d1e1e05afd5b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:50.472846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625ed06b-53c9-419e-a554-d1e1e05afd5b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:50.473114 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625ed06b-53c9-419e-a554-d1e1e05afd5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:50.478354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f8c223-e0e7-4073-97a4-03647ce7c125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:50.478987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f8c223-e0e7-4073-97a4-03647ce7c125 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:50.479266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f8c223-e0e7-4073-97a4-03647ce7c125 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:50.479540 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f8c223-e0e7-4073-97a4-03647ce7c125 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:50.490436 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f8c223-e0e7-4073-97a4-03647ce7c125 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:50.490882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-625ed06b-53c9-419e-a554-d1e1e05afd5b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:50.491890 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f8c223-e0e7-4073-97a4-03647ce7c125 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:50.492083 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-625ed06b-53c9-419e-a554-d1e1e05afd5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-57f8c223-e0e7-4073-97a4-03647ce7c125 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:50.506828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:50.507699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:50.508546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:50.508546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:50.508546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:50.508546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 236/507] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:50.508755 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f8c223-e0e7-4073-97a4-03647ce7c125 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:50.555900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:50.564634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 272/508] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:50.564634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625ed06b-53c9-419e-a554-d1e1e05afd5b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:50.592920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:50.631164 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f8c223-e0e7-4073-97a4-03647ce7c125 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:50.662895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625ed06b-53c9-419e-a554-d1e1e05afd5b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:50.667424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efef2c6d-b6ed-4ea5-a6f1-6ad9467fd695 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:50.668212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efef2c6d-b6ed-4ea5-a6f1-6ad9467fd695 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:50.668631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efef2c6d-b6ed-4ea5-a6f1-6ad9467fd695 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:50.669043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efef2c6d-b6ed-4ea5-a6f1-6ad9467fd695 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:50.680671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efef2c6d-b6ed-4ea5-a6f1-6ad9467fd695 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:50.682440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efef2c6d-b6ed-4ea5-a6f1-6ad9467fd695 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:50.697310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368b2953-0f15-43a9-b919-1c752c37f094 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-efef2c6d-b6ed-4ea5-a6f1-6ad9467fd695 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:50.697862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:50.698761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:50.699602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:50.699602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:50.699602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:50.699773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b2953-0f15-43a9-b919-1c752c37f094 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:50.699773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b2953-0f15-43a9-b919-1c752c37f094 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:50.699773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b2953-0f15-43a9-b919-1c752c37f094 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:50.699773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368b2953-0f15-43a9-b919-1c752c37f094 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:50.699773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368b2953-0f15-43a9-b919-1c752c37f094 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:50.700806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368b2953-0f15-43a9-b919-1c752c37f094 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:50.703663 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 237/509] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:50.704321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efef2c6d-b6ed-4ea5-a6f1-6ad9467fd695 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:50.714059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 273/510] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:50.714890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368b2953-0f15-43a9-b919-1c752c37f094 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:50.775852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:50.814345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efef2c6d-b6ed-4ea5-a6f1-6ad9467fd695 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:50.820909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368b2953-0f15-43a9-b919-1c752c37f094 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:50.834221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:50.850769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670fd71d-56b5-4b11-b711-3bc165caab0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:50.851371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670fd71d-56b5-4b11-b711-3bc165caab0c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:50.851646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670fd71d-56b5-4b11-b711-3bc165caab0c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:50.851959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670fd71d-56b5-4b11-b711-3bc165caab0c None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:50.852680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670fd71d-56b5-4b11-b711-3bc165caab0c None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:50.852980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670fd71d-56b5-4b11-b711-3bc165caab0c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:50.854456 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670fd71d-56b5-4b11-b711-3bc165caab0c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:50.854613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4def47c9-f574-4fbc-a9c8-c083208222a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:50.855206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4def47c9-f574-4fbc-a9c8-c083208222a4 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:50.855481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4def47c9-f574-4fbc-a9c8-c083208222a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:50.855747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4def47c9-f574-4fbc-a9c8-c083208222a4 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:50.856374 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4def47c9-f574-4fbc-a9c8-c083208222a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:50.856671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4def47c9-f574-4fbc-a9c8-c083208222a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:50.858023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4def47c9-f574-4fbc-a9c8-c083208222a4 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:50.869449 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 274/511] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:50.870235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4def47c9-f574-4fbc-a9c8-c083208222a4 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:50.876071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 238/512] 10.222.0.65 () {56 vars in 1137 bytes} [Wed Aug 24 03:26:50 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:50.876752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670fd71d-56b5-4b11-b711-3bc165caab0c None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:50.967962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4def47c9-f574-4fbc-a9c8-c083208222a4 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:50.977351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670fd71d-56b5-4b11-b711-3bc165caab0c None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:50.985432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:51.002559 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d932666-7ff9-4976-bd31-d176d203f531 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.003188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d932666-7ff9-4976-bd31-d176d203f531 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.003459 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d932666-7ff9-4976-bd31-d176d203f531 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.003726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d932666-7ff9-4976-bd31-d176d203f531 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.004363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d932666-7ff9-4976-bd31-d176d203f531 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:51.004661 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d932666-7ff9-4976-bd31-d176d203f531 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:51.006329 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d932666-7ff9-4976-bd31-d176d203f531 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:51.010842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e499ed-3203-4919-8853-17a9465f4b01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.011464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e499ed-3203-4919-8853-17a9465f4b01 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.011744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e499ed-3203-4919-8853-17a9465f4b01 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.012015 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e499ed-3203-4919-8853-17a9465f4b01 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.018011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 275/513] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:51.019345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d932666-7ff9-4976-bd31-d176d203f531 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:51.021176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e499ed-3203-4919-8853-17a9465f4b01 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:51.022677 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e499ed-3203-4919-8853-17a9465f4b01 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-74e499ed-3203-4919-8853-17a9465f4b01 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:26:51.035140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:51.036094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:51.036977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:51.036977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:51.036977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:26:51.036977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:26:51.036977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:26:51.036977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:26:51.036977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:26:51.036977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:26:51.036977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 24 03:26:51.036977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:51.036977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 239/514] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:26:50 2022] GET /identity/v3/users/neutron => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:51.037557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e499ed-3203-4919-8853-17a9465f4b01 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:51.070486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:51.123113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d932666-7ff9-4976-bd31-d176d203f531 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:51.132888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e499ed-3203-4919-8853-17a9465f4b01 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:51.156509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120c53dd-d888-4ea9-b257-dc57d4c4d113 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.157137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120c53dd-d888-4ea9-b257-dc57d4c4d113 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.157416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120c53dd-d888-4ea9-b257-dc57d4c4d113 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.157689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120c53dd-d888-4ea9-b257-dc57d4c4d113 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.166677 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb82c48-f3cb-40cb-9f3a-2141a33f9699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.167285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb82c48-f3cb-40cb-9f3a-2141a33f9699 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.167558 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb82c48-f3cb-40cb-9f3a-2141a33f9699 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.167824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb82c48-f3cb-40cb-9f3a-2141a33f9699 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.168558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120c53dd-d888-4ea9-b257-dc57d4c4d113 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:51.170025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120c53dd-d888-4ea9-b257-dc57d4c4d113 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:51.174207 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb82c48-f3cb-40cb-9f3a-2141a33f9699 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:51.175723 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb82c48-f3cb-40cb-9f3a-2141a33f9699 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-120c53dd-d888-4ea9-b257-dc57d4c4d113 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:51.184084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:51.185138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:51.185986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:51.185986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:51.185986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:51.185986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 276/515] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:51.185986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120c53dd-d888-4ea9-b257-dc57d4c4d113 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-cbb82c48-f3cb-40cb-9f3a-2141a33f9699 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:26:51.187726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:51.188560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:51.189422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:26:51.189422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:26:51.189422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:26:51.189422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:26:51.189422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:26:51.189422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:26:51.189422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:51.189422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:51.189422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 240/516] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:51.191024 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb82c48-f3cb-40cb-9f3a-2141a33f9699 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:51.218616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:51.282707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:51.289301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120c53dd-d888-4ea9-b257-dc57d4c4d113 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:51.289437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb82c48-f3cb-40cb-9f3a-2141a33f9699 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:51.323230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06fc9ac-1dc1-4795-91e3-55978302d521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.323429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53baca21-c4c6-4ef0-a0e4-0e124a339fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.323824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53baca21-c4c6-4ef0-a0e4-0e124a339fe5 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.323916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06fc9ac-1dc1-4795-91e3-55978302d521 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.324104 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53baca21-c4c6-4ef0-a0e4-0e124a339fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.324193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06fc9ac-1dc1-4795-91e3-55978302d521 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.324371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06fc9ac-1dc1-4795-91e3-55978302d521 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.324455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53baca21-c4c6-4ef0-a0e4-0e124a339fe5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.331281 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06fc9ac-1dc1-4795-91e3-55978302d521 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:51.332714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06fc9ac-1dc1-4795-91e3-55978302d521 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-a06fc9ac-1dc1-4795-91e3-55978302d521 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:26:51.344187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:51.345101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:51.345995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:51.345995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:51.345995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:26:51.345995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:26:51.345995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:26:51.345995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:26:51.345995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:26:51.345995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:26:51.345995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 03:26:51.345995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:51.345995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 241/517] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:51 2022] GET /identity/v3/users/cinder => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:51.346602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06fc9ac-1dc1-4795-91e3-55978302d521 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:51.396859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53baca21-c4c6-4ef0-a0e4-0e124a339fe5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:51.398256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53baca21-c4c6-4ef0-a0e4-0e124a339fe5 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:51.425062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:51.448925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06fc9ac-1dc1-4795-91e3-55978302d521 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:51.483086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1c3c44-1b54-4647-b89c-687de6e489bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.483692 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1c3c44-1b54-4647-b89c-687de6e489bf None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.483974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1c3c44-1b54-4647-b89c-687de6e489bf None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.484246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1c3c44-1b54-4647-b89c-687de6e489bf None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.484974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1c3c44-1b54-4647-b89c-687de6e489bf None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:51.485274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1c3c44-1b54-4647-b89c-687de6e489bf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:51.486758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1c3c44-1b54-4647-b89c-687de6e489bf None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:51.497094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 277/518] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:51.497953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53baca21-c4c6-4ef0-a0e4-0e124a339fe5 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:51.506050 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 242/519] 10.222.0.65 () {56 vars in 1139 bytes} [Wed Aug 24 03:26:51 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:51.506888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1c3c44-1b54-4647-b89c-687de6e489bf None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:51.522794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:51.601528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53baca21-c4c6-4ef0-a0e4-0e124a339fe5 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:51.604474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1c3c44-1b54-4647-b89c-687de6e489bf None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:51.627155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:51.635755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6a4021-5b30-4aa1-b43a-d7c358e571e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.636354 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6a4021-5b30-4aa1-b43a-d7c358e571e2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.636634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6a4021-5b30-4aa1-b43a-d7c358e571e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.636901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6a4021-5b30-4aa1-b43a-d7c358e571e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.637535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6a4021-5b30-4aa1-b43a-d7c358e571e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:51.637833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6a4021-5b30-4aa1-b43a-d7c358e571e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:51.637918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3c3c2f-e91b-4992-9bcf-99b15d09fcf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.638541 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3c3c2f-e91b-4992-9bcf-99b15d09fcf5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.638817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3c3c2f-e91b-4992-9bcf-99b15d09fcf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.639085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3c3c2f-e91b-4992-9bcf-99b15d09fcf5 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.639236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6a4021-5b30-4aa1-b43a-d7c358e571e2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:51.639759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3c3c2f-e91b-4992-9bcf-99b15d09fcf5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:51.640061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3c3c2f-e91b-4992-9bcf-99b15d09fcf5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:51.641476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3c3c2f-e91b-4992-9bcf-99b15d09fcf5 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:51.650587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 278/520] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:51.651333 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6a4021-5b30-4aa1-b43a-d7c358e571e2 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:51.662620 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 243/521] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:26:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:51.663633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3c3c2f-e91b-4992-9bcf-99b15d09fcf5 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:51.738544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:51.758229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6a4021-5b30-4aa1-b43a-d7c358e571e2 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:51.764792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3c3c2f-e91b-4992-9bcf-99b15d09fcf5 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:51.792118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7694a42-204d-4b2c-a805-75f1e26bf8ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.792719 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7694a42-204d-4b2c-a805-75f1e26bf8ee None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.792996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7694a42-204d-4b2c-a805-75f1e26bf8ee None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.793263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7694a42-204d-4b2c-a805-75f1e26bf8ee None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.793986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7694a42-204d-4b2c-a805-75f1e26bf8ee None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:51.794319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7694a42-204d-4b2c-a805-75f1e26bf8ee None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:51.795753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7694a42-204d-4b2c-a805-75f1e26bf8ee None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:51.800493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbe7ab7-2ddb-42e2-a26b-7bde29f7189c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.801399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbe7ab7-2ddb-42e2-a26b-7bde29f7189c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.801671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbe7ab7-2ddb-42e2-a26b-7bde29f7189c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.801939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbe7ab7-2ddb-42e2-a26b-7bde29f7189c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.813269 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbe7ab7-2ddb-42e2-a26b-7bde29f7189c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:51.814827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbe7ab7-2ddb-42e2-a26b-7bde29f7189c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-bbbe7ab7-2ddb-42e2-a26b-7bde29f7189c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:51.829367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:51.830326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:51.831490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:51.831490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:51.831490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:51.831490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 244/522] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:51.832086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbe7ab7-2ddb-42e2-a26b-7bde29f7189c None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:51.832689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 279/523] 10.222.0.65 () {56 vars in 1137 bytes} [Wed Aug 24 03:26:51 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:51.833780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7694a42-204d-4b2c-a805-75f1e26bf8ee None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:51.879706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:51.932317 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbe7ab7-2ddb-42e2-a26b-7bde29f7189c None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:51.933345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7694a42-204d-4b2c-a805-75f1e26bf8ee None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:51.957273 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:51.965524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a1c7a2-0be8-4655-92ce-55b5b0f95a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.966122 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a1c7a2-0be8-4655-92ce-55b5b0f95a1b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.966428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a1c7a2-0be8-4655-92ce-55b5b0f95a1b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.966702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a1c7a2-0be8-4655-92ce-55b5b0f95a1b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.967935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7177572e-0b6a-4d33-9858-23f300108f62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:51.968528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7177572e-0b6a-4d33-9858-23f300108f62 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:51.969045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7177572e-0b6a-4d33-9858-23f300108f62 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:51.969224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7177572e-0b6a-4d33-9858-23f300108f62 None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:51.969871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7177572e-0b6a-4d33-9858-23f300108f62 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:51.971331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7177572e-0b6a-4d33-9858-23f300108f62 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:52.032897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a1c7a2-0be8-4655-92ce-55b5b0f95a1b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:52.034901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a1c7a2-0be8-4655-92ce-55b5b0f95a1b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:52.092902 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 280/524] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:26:51 2022] POST /identity/v3/users => generated 247 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:26:52.093921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7177572e-0b6a-4d33-9858-23f300108f62 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:52.145056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 245/525] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:52.145056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a1c7a2-0be8-4655-92ce-55b5b0f95a1b None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:52.173247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:52.208371 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7177572e-0b6a-4d33-9858-23f300108f62 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:52.210488 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:52.255156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfdf286-73b9-41af-bf1d-f22055b36b55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:52.256022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfdf286-73b9-41af-bf1d-f22055b36b55 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:52.256326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfdf286-73b9-41af-bf1d-f22055b36b55 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:52.256600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfdf286-73b9-41af-bf1d-f22055b36b55 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:52.257777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfdf286-73b9-41af-bf1d-f22055b36b55 None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=e5896018712d4b82a781365e478ee855) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:52.257777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfdf286-73b9-41af-bf1d-f22055b36b55 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:52.260182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfdf286-73b9-41af-bf1d-f22055b36b55 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:52.262306 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a1c7a2-0be8-4655-92ce-55b5b0f95a1b None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:52.265782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 281/526] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:52 2022] GET /identity/v3/role_assignments?user.id=e5896018712d4b82a781365e478ee855&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 259 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:52.266853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfdf286-73b9-41af-bf1d-f22055b36b55 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:52.312579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d5f9f5-c1f7-4bad-a780-aaac176c3c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:52.313226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d5f9f5-c1f7-4bad-a780-aaac176c3c85 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:52.313501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d5f9f5-c1f7-4bad-a780-aaac176c3c85 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:52.313767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d5f9f5-c1f7-4bad-a780-aaac176c3c85 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:52.314466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d5f9f5-c1f7-4bad-a780-aaac176c3c85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:52.314764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d5f9f5-c1f7-4bad-a780-aaac176c3c85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:52.316390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d5f9f5-c1f7-4bad-a780-aaac176c3c85 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:52.330929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 246/527] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:52.332036 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d5f9f5-c1f7-4bad-a780-aaac176c3c85 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:52.418113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfdf286-73b9-41af-bf1d-f22055b36b55 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:52.452323 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:52.458717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d84b054-05a8-4bc8-832d-d5e8b2a26206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:52.459406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d84b054-05a8-4bc8-832d-d5e8b2a26206 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:52.459787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d84b054-05a8-4bc8-832d-d5e8b2a26206 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:52.460155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d84b054-05a8-4bc8-832d-d5e8b2a26206 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:52.473156 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d5f9f5-c1f7-4bad-a780-aaac176c3c85 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:52.504970 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:52.521825 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e68e95-9612-4901-b54b-686630aaa9e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:52.523100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e68e95-9612-4901-b54b-686630aaa9e4 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:52.523711 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e68e95-9612-4901-b54b-686630aaa9e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:52.524482 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e68e95-9612-4901-b54b-686630aaa9e4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:52.535455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d84b054-05a8-4bc8-832d-d5e8b2a26206 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:52.537328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d84b054-05a8-4bc8-832d-d5e8b2a26206 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:52.539646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e68e95-9612-4901-b54b-686630aaa9e4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:52.542231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e68e95-9612-4901-b54b-686630aaa9e4 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-d0e68e95-9612-4901-b54b-686630aaa9e4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:52.562380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:52.563355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:52.564206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:52.564206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:52.564206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:52.567527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 247/528] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:52.568805 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e68e95-9612-4901-b54b-686630aaa9e4 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:52.665995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 282/529] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:52.667287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d84b054-05a8-4bc8-832d-d5e8b2a26206 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:52.684838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e68e95-9612-4901-b54b-686630aaa9e4 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:52.722071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7265f8aa-ed1f-45f6-9b92-04a941b7ebd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:52.722795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7265f8aa-ed1f-45f6-9b92-04a941b7ebd9 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:52.723073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7265f8aa-ed1f-45f6-9b92-04a941b7ebd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:52.723344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7265f8aa-ed1f-45f6-9b92-04a941b7ebd9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:52.730609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7265f8aa-ed1f-45f6-9b92-04a941b7ebd9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:52.732330 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7265f8aa-ed1f-45f6-9b92-04a941b7ebd9 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:52.733945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:52.744080 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-7265f8aa-ed1f-45f6-9b92-04a941b7ebd9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:26:52.746086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:52.746963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:52.747808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:26:52.747808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:26:52.747808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:26:52.747808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:26:52.747808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:26:52.747808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:26:52.747808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:52.747808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:52.747808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 248/530] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:52.748293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7265f8aa-ed1f-45f6-9b92-04a941b7ebd9 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:52.791766 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d84b054-05a8-4bc8-832d-d5e8b2a26206 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:52.829299 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661f4839-8e5b-4167-94b9-176e8ca00625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:52.830445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661f4839-8e5b-4167-94b9-176e8ca00625 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:52.831082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661f4839-8e5b-4167-94b9-176e8ca00625 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:52.831646 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661f4839-8e5b-4167-94b9-176e8ca00625 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:52.832596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661f4839-8e5b-4167-94b9-176e8ca00625 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:52.833096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661f4839-8e5b-4167-94b9-176e8ca00625 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:52.834881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661f4839-8e5b-4167-94b9-176e8ca00625 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:52.848900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 283/531] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:52.850026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661f4839-8e5b-4167-94b9-176e8ca00625 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:52.854793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7265f8aa-ed1f-45f6-9b92-04a941b7ebd9 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:52.890795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727ac0e4-09b0-4aa9-b406-dc79b7befa45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:52.891421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727ac0e4-09b0-4aa9-b406-dc79b7befa45 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:52.891695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727ac0e4-09b0-4aa9-b406-dc79b7befa45 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:52.891961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727ac0e4-09b0-4aa9-b406-dc79b7befa45 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:52.903746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727ac0e4-09b0-4aa9-b406-dc79b7befa45 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:52.905290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727ac0e4-09b0-4aa9-b406-dc79b7befa45 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-727ac0e4-09b0-4aa9-b406-dc79b7befa45 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:52.920729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:52.921654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:52.922908 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:52.922908 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:52.922908 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:52.922908 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 249/532] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:52.923141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727ac0e4-09b0-4aa9-b406-dc79b7befa45 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:52.964069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661f4839-8e5b-4167-94b9-176e8ca00625 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:52.985902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:52.992275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:53.001850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a4c8f0-a8b8-4b9c-8fed-2c9d6e639c44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:53.003786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a4c8f0-a8b8-4b9c-8fed-2c9d6e639c44 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:53.004290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a4c8f0-a8b8-4b9c-8fed-2c9d6e639c44 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:53.005035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a4c8f0-a8b8-4b9c-8fed-2c9d6e639c44 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:53.006202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a4c8f0-a8b8-4b9c-8fed-2c9d6e639c44 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:53.006920 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a4c8f0-a8b8-4b9c-8fed-2c9d6e639c44 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:53.009342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a4c8f0-a8b8-4b9c-8fed-2c9d6e639c44 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:53.038105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 284/533] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:26:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:53.038105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a4c8f0-a8b8-4b9c-8fed-2c9d6e639c44 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:53.049361 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727ac0e4-09b0-4aa9-b406-dc79b7befa45 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:53.087410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5d75f9-c80e-4dbf-bfc5-ec4cca8259f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:53.088095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5d75f9-c80e-4dbf-bfc5-ec4cca8259f8 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:53.088377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5d75f9-c80e-4dbf-bfc5-ec4cca8259f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:53.088651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5d75f9-c80e-4dbf-bfc5-ec4cca8259f8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:53.095857 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5d75f9-c80e-4dbf-bfc5-ec4cca8259f8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:53.097953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5d75f9-c80e-4dbf-bfc5-ec4cca8259f8 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-5a5d75f9-c80e-4dbf-bfc5-ec4cca8259f8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:26:53.109828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:53.110734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:53.111591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:26:53.111591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:26:53.111591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:26:53.111591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:26:53.111591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:26:53.111591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:26:53.111591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:53.111591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:53.111591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 250/534] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:53.111591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5d75f9-c80e-4dbf-bfc5-ec4cca8259f8 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:53.170349 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a4c8f0-a8b8-4b9c-8fed-2c9d6e639c44 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:53.208051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e5e5e6-5bf1-4124-ba4c-122b97e0ac4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:53.209133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e5e5e6-5bf1-4124-ba4c-122b97e0ac4f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:53.209647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e5e5e6-5bf1-4124-ba4c-122b97e0ac4f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:53.210113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e5e5e6-5bf1-4124-ba4c-122b97e0ac4f None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:53.210978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e5e5e6-5bf1-4124-ba4c-122b97e0ac4f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:53.211489 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e5e5e6-5bf1-4124-ba4c-122b97e0ac4f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:53.213342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e5e5e6-5bf1-4124-ba4c-122b97e0ac4f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:53.221449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5d75f9-c80e-4dbf-bfc5-ec4cca8259f8 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:53.227081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 285/535] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:53.228214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e5e5e6-5bf1-4124-ba4c-122b97e0ac4f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:53.233637 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:53.259830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3273b90f-ad14-40f7-9a03-4a676d8a272c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:53.259830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3273b90f-ad14-40f7-9a03-4a676d8a272c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:53.259830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3273b90f-ad14-40f7-9a03-4a676d8a272c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:53.259830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3273b90f-ad14-40f7-9a03-4a676d8a272c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:53.260378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3273b90f-ad14-40f7-9a03-4a676d8a272c None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=9f3b3c7ebc08496e970f8f25fd6395e4) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:53.260603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3273b90f-ad14-40f7-9a03-4a676d8a272c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:53.262034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3273b90f-ad14-40f7-9a03-4a676d8a272c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:53.274178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 251/536] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:53 2022] GET /identity/v3/role_assignments?user.id=9f3b3c7ebc08496e970f8f25fd6395e4&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:53.274178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3273b90f-ad14-40f7-9a03-4a676d8a272c None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:53.296624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:53.383753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3273b90f-ad14-40f7-9a03-4a676d8a272c None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:53.426483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5cbf79-e0ca-4a87-b431-42dae238fca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:53.427136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5cbf79-e0ca-4a87-b431-42dae238fca5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:53.427410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5cbf79-e0ca-4a87-b431-42dae238fca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:53.427677 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5cbf79-e0ca-4a87-b431-42dae238fca5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:53.441277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5cbf79-e0ca-4a87-b431-42dae238fca5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:53.442786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5cbf79-e0ca-4a87-b431-42dae238fca5 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:53.446073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e5e5e6-5bf1-4124-ba4c-122b97e0ac4f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-ea5cbf79-e0ca-4a87-b431-42dae238fca5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:26:53.468561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:53.469481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:53.470350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:26:53.470350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:26:53.470350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:26:53.470350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:26:53.470350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:26:53.470350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:26:53.470350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:53.470350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:53.470350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 252/537] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:53.477005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5cbf79-e0ca-4a87-b431-42dae238fca5 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:53.510538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccdea77-7103-456c-8b0c-ca43800784dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:53.511075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccdea77-7103-456c-8b0c-ca43800784dc None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:53.511364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccdea77-7103-456c-8b0c-ca43800784dc None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:53.511670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccdea77-7103-456c-8b0c-ca43800784dc None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:53.512361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccdea77-7103-456c-8b0c-ca43800784dc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:53.512657 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccdea77-7103-456c-8b0c-ca43800784dc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:53.514595 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccdea77-7103-456c-8b0c-ca43800784dc None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:53.528795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:53.542036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 286/538] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:26:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:53.543290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccdea77-7103-456c-8b0c-ca43800784dc None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:53.661114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5cbf79-e0ca-4a87-b431-42dae238fca5 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:53.709920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a43ee6d-5add-46a0-ac78-29f9b4a14d32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:53.710898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a43ee6d-5add-46a0-ac78-29f9b4a14d32 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:53.711371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a43ee6d-5add-46a0-ac78-29f9b4a14d32 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:53.711837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a43ee6d-5add-46a0-ac78-29f9b4a14d32 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:53.712748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a43ee6d-5add-46a0-ac78-29f9b4a14d32 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:53.713277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a43ee6d-5add-46a0-ac78-29f9b4a14d32 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:53.714987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a43ee6d-5add-46a0-ac78-29f9b4a14d32 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:53.727506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccdea77-7103-456c-8b0c-ca43800784dc None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:53.729455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 253/539] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:26:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:53.739231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:53.747166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a43ee6d-5add-46a0-ac78-29f9b4a14d32 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:53.774912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fafec30-8eb4-4f90-9d95-85c745c99894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:53.775557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fafec30-8eb4-4f90-9d95-85c745c99894 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:53.775837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fafec30-8eb4-4f90-9d95-85c745c99894 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:53.776107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fafec30-8eb4-4f90-9d95-85c745c99894 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:53.776838 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fafec30-8eb4-4f90-9d95-85c745c99894 None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=64f23f0c8abc406b9700361723623387) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:53.777142 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fafec30-8eb4-4f90-9d95-85c745c99894 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:53.778761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fafec30-8eb4-4f90-9d95-85c745c99894 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:53.792587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 287/540] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:53 2022] GET /identity/v3/role_assignments?user.id=64f23f0c8abc406b9700361723623387&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 259 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:53.871116 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:53.878995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a43ee6d-5add-46a0-ac78-29f9b4a14d32 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:53.933435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ab5151-e6af-4da4-851f-0053c9196c63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:53.934069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ab5151-e6af-4da4-851f-0053c9196c63 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:53.934377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ab5151-e6af-4da4-851f-0053c9196c63 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:53.934648 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ab5151-e6af-4da4-851f-0053c9196c63 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:53.935401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ab5151-e6af-4da4-851f-0053c9196c63 None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=dbfbfd64573141f88040692119ac5a1f) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:53.935707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ab5151-e6af-4da4-851f-0053c9196c63 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:53.937166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ab5151-e6af-4da4-851f-0053c9196c63 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:53.946919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 254/541] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:53 2022] GET /identity/v3/role_assignments?user.id=dbfbfd64573141f88040692119ac5a1f&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 259 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:54.326511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78c484f-072f-4f17-9d15-1d373cb7d7db None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:54.327052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78c484f-072f-4f17-9d15-1d373cb7d7db None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:54.327052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78c484f-072f-4f17-9d15-1d373cb7d7db None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:54.328615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 288/542] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:54.334512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a86eb66-09cf-49cf-b942-3733a0a293a0 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:54.334959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a86eb66-09cf-49cf-b942-3733a0a293a0 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:54.335046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a86eb66-09cf-49cf-b942-3733a0a293a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:54.353725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a40c62-8aea-4ae2-a00d-70e93853ba0b None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:54.354026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a40c62-8aea-4ae2-a00d-70e93853ba0b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:54.354299 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a40c62-8aea-4ae2-a00d-70e93853ba0b None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:54.358040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 289/543] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:54 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:54.363217 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cf4671-0762-4432-9328-eb48718f41fe None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:54.363765 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cf4671-0762-4432-9328-eb48718f41fe None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:54.364411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cf4671-0762-4432-9328-eb48718f41fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:54.368417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-4a86eb66-09cf-49cf-b942-3733a0a293a0 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:54.369939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4a86eb66-09cf-49cf-b942-3733a0a293a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:54.387248 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d3cf4671-0762-4432-9328-eb48718f41fe None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:54.388121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d3cf4671-0762-4432-9328-eb48718f41fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:54.463080 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 290/544] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:54.517930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 255/545] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:26:54.762478 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cf4671-0762-4432-9328-eb48718f41fe None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:54.879268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:54.882831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cf4671-0762-4432-9328-eb48718f41fe None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:54.915102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a86eb66-09cf-49cf-b942-3733a0a293a0 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:54.941392 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb47ebd-e5d6-42bb-80d2-56de2ef5d97f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:54.942026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb47ebd-e5d6-42bb-80d2-56de2ef5d97f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:54.942324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb47ebd-e5d6-42bb-80d2-56de2ef5d97f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:54.942590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb47ebd-e5d6-42bb-80d2-56de2ef5d97f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:54.971034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb47ebd-e5d6-42bb-80d2-56de2ef5d97f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-0eb47ebd-e5d6-42bb-80d2-56de2ef5d97f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:54.980119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:54.981215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:54.982139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 291/546] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:26:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:54.989441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb47ebd-e5d6-42bb-80d2-56de2ef5d97f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:55.026881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:55.041685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a86eb66-09cf-49cf-b942-3733a0a293a0 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:55.080412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee37e848-3792-404a-b473-be1e3972d011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:55.081108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee37e848-3792-404a-b473-be1e3972d011 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:55.081587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee37e848-3792-404a-b473-be1e3972d011 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:55.081872 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee37e848-3792-404a-b473-be1e3972d011 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:55.093105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee37e848-3792-404a-b473-be1e3972d011 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-ee37e848-3792-404a-b473-be1e3972d011 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:55.099038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:55.099913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:55.100781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 256/547] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:26:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:55.105924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee37e848-3792-404a-b473-be1e3972d011 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:55.216326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee37e848-3792-404a-b473-be1e3972d011 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:55.241276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb47ebd-e5d6-42bb-80d2-56de2ef5d97f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:55.278112 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80621718-333a-4b4b-95ae-55a0d764e87b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:55.278656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80621718-333a-4b4b-95ae-55a0d764e87b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:55.279172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80621718-333a-4b4b-95ae-55a0d764e87b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:55.279172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80621718-333a-4b4b-95ae-55a0d764e87b None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:55.279834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80621718-333a-4b4b-95ae-55a0d764e87b None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:55.280130 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80621718-333a-4b4b-95ae-55a0d764e87b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:55.281492 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80621718-333a-4b4b-95ae-55a0d764e87b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:55.283484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daabbd1-55bf-43d6-9bd9-e4c7f386f5f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:55.284602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daabbd1-55bf-43d6-9bd9-e4c7f386f5f6 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:55.284889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daabbd1-55bf-43d6-9bd9-e4c7f386f5f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:55.285142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daabbd1-55bf-43d6-9bd9-e4c7f386f5f6 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:55.285949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daabbd1-55bf-43d6-9bd9-e4c7f386f5f6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:55.286269 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daabbd1-55bf-43d6-9bd9-e4c7f386f5f6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:55.287632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daabbd1-55bf-43d6-9bd9-e4c7f386f5f6 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:55.301285 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 292/548] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:26:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:55.303349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 257/549] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:26:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:55.308193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80621718-333a-4b4b-95ae-55a0d764e87b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:55.309385 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daabbd1-55bf-43d6-9bd9-e4c7f386f5f6 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:55.333280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:55.355068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:55.462267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daabbd1-55bf-43d6-9bd9-e4c7f386f5f6 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:55.506528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9903fe1e-a8db-4790-9cd9-a59c991c18b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:55.508029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9903fe1e-a8db-4790-9cd9-a59c991c18b7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:55.508488 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9903fe1e-a8db-4790-9cd9-a59c991c18b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:55.508960 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9903fe1e-a8db-4790-9cd9-a59c991c18b7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:55.533175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9903fe1e-a8db-4790-9cd9-a59c991c18b7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:55.535067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9903fe1e-a8db-4790-9cd9-a59c991c18b7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:55.539942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80621718-333a-4b4b-95ae-55a0d764e87b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-9903fe1e-a8db-4790-9cd9-a59c991c18b7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:55.568853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:55.569879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:55.570796 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:55.570796 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:55.570796 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:55.573059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 258/550] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:55.575033 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c595f3ef-4a42-4cf8-a55e-672485d202af None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:55.575465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c595f3ef-4a42-4cf8-a55e-672485d202af None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:55.575895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c595f3ef-4a42-4cf8-a55e-672485d202af None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:55.578191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 259/551] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:55.579030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c595f3ef-4a42-4cf8-a55e-672485d202af None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:55.591175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300f1d3f-f491-402d-844f-69ac1afc884d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:55.592158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300f1d3f-f491-402d-844f-69ac1afc884d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:55.592667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300f1d3f-f491-402d-844f-69ac1afc884d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:55.593081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300f1d3f-f491-402d-844f-69ac1afc884d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:55.605297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300f1d3f-f491-402d-844f-69ac1afc884d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:55.607312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300f1d3f-f491-402d-844f-69ac1afc884d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-300f1d3f-f491-402d-844f-69ac1afc884d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:55.623609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:55.624667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:55.625554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:55.625554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:55.625554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:55.627510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 293/552] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:55.633562 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a47ae-56e3-42dd-b545-d1981705894c None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:55.633846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a47ae-56e3-42dd-b545-d1981705894c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:55.634183 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a47ae-56e3-42dd-b545-d1981705894c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:55.651297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:55.655395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-ac8a47ae-56e3-42dd-b545-d1981705894c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:55.657346 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ac8a47ae-56e3-42dd-b545-d1981705894c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:55.694104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c595f3ef-4a42-4cf8-a55e-672485d202af None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:55.729692 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42833ed-462f-4798-9098-0e6aa2345a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:55.730696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42833ed-462f-4798-9098-0e6aa2345a87 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:55.731156 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42833ed-462f-4798-9098-0e6aa2345a87 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:55.731611 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42833ed-462f-4798-9098-0e6aa2345a87 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:55.731881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 294/553] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:55.732694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42833ed-462f-4798-9098-0e6aa2345a87 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:55.732800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a47ae-56e3-42dd-b545-d1981705894c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:55.733298 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42833ed-462f-4798-9098-0e6aa2345a87 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:55.734928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42833ed-462f-4798-9098-0e6aa2345a87 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:55.735953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:55.749400 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 260/554] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:55.756273 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42833ed-462f-4798-9098-0e6aa2345a87 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:55.842578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a47ae-56e3-42dd-b545-d1981705894c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:55.881418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c791805-48c9-46fb-a564-2dddc639c97e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:55.882034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c791805-48c9-46fb-a564-2dddc639c97e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:55.882382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c791805-48c9-46fb-a564-2dddc639c97e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:55.882657 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c791805-48c9-46fb-a564-2dddc639c97e None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:55.883303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c791805-48c9-46fb-a564-2dddc639c97e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:55.883601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c791805-48c9-46fb-a564-2dddc639c97e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:55.884966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c791805-48c9-46fb-a564-2dddc639c97e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:55.897418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 295/555] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:55.902718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c791805-48c9-46fb-a564-2dddc639c97e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:55.970993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:56.013710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c791805-48c9-46fb-a564-2dddc639c97e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:56.030537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42833ed-462f-4798-9098-0e6aa2345a87 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:56.053436 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fc1872-edc1-4009-80c3-d81b7d8335e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:56.054123 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fc1872-edc1-4009-80c3-d81b7d8335e2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.054431 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fc1872-edc1-4009-80c3-d81b7d8335e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.054878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fc1872-edc1-4009-80c3-d81b7d8335e2 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.060248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:56.064647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fc1872-edc1-4009-80c3-d81b7d8335e2 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:56.066867 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fc1872-edc1-4009-80c3-d81b7d8335e2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-f1fc1872-edc1-4009-80c3-d81b7d8335e2 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:26:56.079087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:56.080065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:56.081121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:56.081121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:56.081121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:26:56.081121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:26:56.081121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:26:56.081121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:26:56.081121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:26:56.081121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:26:56.081121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:26:56.081121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:56.081121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 296/556] 10.222.0.65 () {56 vars in 1093 bytes} [Wed Aug 24 03:26:55 2022] GET /identity/v3/users/nova => generated 82 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:56.082114 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79985252-7e45-4e13-ade2-16dcd48c337d None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.086113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79985252-7e45-4e13-ade2-16dcd48c337d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.087706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79985252-7e45-4e13-ade2-16dcd48c337d None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.089160 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 297/557] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:56 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:56.090265 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79985252-7e45-4e13-ade2-16dcd48c337d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:56.096557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045edc0d-b6e9-430c-a2fc-ef7376640cc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:56.097822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045edc0d-b6e9-430c-a2fc-ef7376640cc2 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.102522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045edc0d-b6e9-430c-a2fc-ef7376640cc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.102803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045edc0d-b6e9-430c-a2fc-ef7376640cc2 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.113059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045edc0d-b6e9-430c-a2fc-ef7376640cc2 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:56.115017 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045edc0d-b6e9-430c-a2fc-ef7376640cc2 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-045edc0d-b6e9-430c-a2fc-ef7376640cc2 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:26:56.130222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:56.131494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:56.132931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:56.132931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:56.132931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:26:56.132931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:26:56.132931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:26:56.132931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:26:56.132931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:26:56.132931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:26:56.132931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 24 03:26:56.132931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:56.132931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 261/558] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:26:55 2022] GET /identity/v3/users/swift => generated 83 bytes in 376 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:26:56.132931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85544db5-4fec-4a46-a1ba-dfe2211fd01d None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.133623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85544db5-4fec-4a46-a1ba-dfe2211fd01d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.133623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85544db5-4fec-4a46-a1ba-dfe2211fd01d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.152459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-85544db5-4fec-4a46-a1ba-dfe2211fd01d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:56.153504 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-85544db5-4fec-4a46-a1ba-dfe2211fd01d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:56.210416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79985252-7e45-4e13-ade2-16dcd48c337d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:56.228032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 262/559] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:56.229094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85544db5-4fec-4a46-a1ba-dfe2211fd01d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:56.235991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:56.249822 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b001cd5-2b14-45b1-bb7a-dbadc4de5f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:56.250568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b001cd5-2b14-45b1-bb7a-dbadc4de5f2a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.251064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b001cd5-2b14-45b1-bb7a-dbadc4de5f2a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.251519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b001cd5-2b14-45b1-bb7a-dbadc4de5f2a None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.252486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b001cd5-2b14-45b1-bb7a-dbadc4de5f2a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:56.253182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b001cd5-2b14-45b1-bb7a-dbadc4de5f2a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:56.255207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b001cd5-2b14-45b1-bb7a-dbadc4de5f2a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:56.280846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 298/560] 10.222.0.65 () {56 vars in 1133 bytes} [Wed Aug 24 03:26:56 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:56.287107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b001cd5-2b14-45b1-bb7a-dbadc4de5f2a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:56.352044 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85544db5-4fec-4a46-a1ba-dfe2211fd01d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:56.366415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:56.389099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf3ddb9-4054-43ba-9dd2-c8e3d4d0aa92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:56.390071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf3ddb9-4054-43ba-9dd2-c8e3d4d0aa92 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.390664 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf3ddb9-4054-43ba-9dd2-c8e3d4d0aa92 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.391132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf3ddb9-4054-43ba-9dd2-c8e3d4d0aa92 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.395231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b001cd5-2b14-45b1-bb7a-dbadc4de5f2a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:56.404156 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf3ddb9-4054-43ba-9dd2-c8e3d4d0aa92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-8cf3ddb9-4054-43ba-9dd2-c8e3d4d0aa92 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:56.409876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:56.410997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:56.411849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:56.411849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:56.411849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:56.411849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:56.411849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:56.411849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:56.411849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:56.411849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:56.411849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:56.411849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:56.411849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:56.411849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:56.415170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 263/561] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:26:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:56.416229 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf3ddb9-4054-43ba-9dd2-c8e3d4d0aa92 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:56.442601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20880635-fd17-45bb-b945-ee153eccf51e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:56.443280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20880635-fd17-45bb-b945-ee153eccf51e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.443483 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20880635-fd17-45bb-b945-ee153eccf51e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.443757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20880635-fd17-45bb-b945-ee153eccf51e None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.444492 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20880635-fd17-45bb-b945-ee153eccf51e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:56.444795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20880635-fd17-45bb-b945-ee153eccf51e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:56.446339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20880635-fd17-45bb-b945-ee153eccf51e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:56.464798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 299/562] 10.222.0.65 () {56 vars in 1135 bytes} [Wed Aug 24 03:26:56 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:56.465624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20880635-fd17-45bb-b945-ee153eccf51e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:56.516005 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:56.546558 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf3ddb9-4054-43ba-9dd2-c8e3d4d0aa92 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:56.575060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20880635-fd17-45bb-b945-ee153eccf51e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:56.580131 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9697ea4f-0bfe-406e-9a5f-9157e97375c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:56.580757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9697ea4f-0bfe-406e-9a5f-9157e97375c2 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.581613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9697ea4f-0bfe-406e-9a5f-9157e97375c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.582053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9697ea4f-0bfe-406e-9a5f-9157e97375c2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.592255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9697ea4f-0bfe-406e-9a5f-9157e97375c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:56.593816 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9697ea4f-0bfe-406e-9a5f-9157e97375c2 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-9697ea4f-0bfe-406e-9a5f-9157e97375c2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:56.608000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:56.608985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:56.609866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:56.609866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:56.609866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:56.610043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7088f4-6877-4130-88c9-1be47a4b4c41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:56.610043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7088f4-6877-4130-88c9-1be47a4b4c41 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.610043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7088f4-6877-4130-88c9-1be47a4b4c41 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.610282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 264/563] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:56.610352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7088f4-6877-4130-88c9-1be47a4b4c41 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.610352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7088f4-6877-4130-88c9-1be47a4b4c41 None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:56.610490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9697ea4f-0bfe-406e-9a5f-9157e97375c2 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:56.610558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7088f4-6877-4130-88c9-1be47a4b4c41 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:56.611437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7088f4-6877-4130-88c9-1be47a4b4c41 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:56.619679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 300/564] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:26:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:56.620911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5b6552-70fe-4cba-915a-a70c1444a41e None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.621199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5b6552-70fe-4cba-915a-a70c1444a41e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.621467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5b6552-70fe-4cba-915a-a70c1444a41e None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.626420 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 301/565] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:26:56 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:26:56.627137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5b6552-70fe-4cba-915a-a70c1444a41e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:56.638071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:56.711377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9697ea4f-0bfe-406e-9a5f-9157e97375c2 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:56.728493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5b6552-70fe-4cba-915a-a70c1444a41e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:56.730118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:56.744917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a9c677-e251-4cec-b77d-d41f8ab40a95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:56.745529 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a9c677-e251-4cec-b77d-d41f8ab40a95 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.745808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a9c677-e251-4cec-b77d-d41f8ab40a95 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.746080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a9c677-e251-4cec-b77d-d41f8ab40a95 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.762747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba11c0e-94d4-4242-8b48-895855c7039b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:56.763367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba11c0e-94d4-4242-8b48-895855c7039b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.763645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba11c0e-94d4-4242-8b48-895855c7039b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.763919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba11c0e-94d4-4242-8b48-895855c7039b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.774584 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba11c0e-94d4-4242-8b48-895855c7039b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-eba11c0e-94d4-4242-8b48-895855c7039b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:56.779609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:56.780616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:56.781455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 302/566] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:26:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:56.782212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba11c0e-94d4-4242-8b48-895855c7039b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:56.820115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a9c677-e251-4cec-b77d-d41f8ab40a95 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:56.821610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a9c677-e251-4cec-b77d-d41f8ab40a95 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:56.845065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:56.879700 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba11c0e-94d4-4242-8b48-895855c7039b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:56.912919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b81b070-44ed-45ee-8945-4badbf8c8a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:56.913515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81b070-44ed-45ee-8945-4badbf8c8a69 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.913799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81b070-44ed-45ee-8945-4badbf8c8a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.914073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81b070-44ed-45ee-8945-4badbf8c8a69 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.914750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b81b070-44ed-45ee-8945-4badbf8c8a69 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:56.915052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b81b070-44ed-45ee-8945-4badbf8c8a69 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:56.916462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b81b070-44ed-45ee-8945-4badbf8c8a69 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:56.921003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 265/567] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:26:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:26:56.921835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a9c677-e251-4cec-b77d-d41f8ab40a95 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:56.931472 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 303/568] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:56.932557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc5360b-cdbc-4574-bee3-22ef15ac3f94 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:56.932844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc5360b-cdbc-4574-bee3-22ef15ac3f94 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:56.933120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc5360b-cdbc-4574-bee3-22ef15ac3f94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:56.950905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-edc5360b-cdbc-4574-bee3-22ef15ac3f94 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:26:56.952099 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-edc5360b-cdbc-4574-bee3-22ef15ac3f94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:26:56.967998 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:57.018875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 304/569] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:26:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:26:57.019771 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc5360b-cdbc-4574-bee3-22ef15ac3f94 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.039671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a9c677-e251-4cec-b77d-d41f8ab40a95 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:57.065075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:57.076487 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb99eb1-6386-46fe-9962-f3235cadab1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:57.077118 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb99eb1-6386-46fe-9962-f3235cadab1f None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:57.077402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb99eb1-6386-46fe-9962-f3235cadab1f None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:57.077671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb99eb1-6386-46fe-9962-f3235cadab1f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:57.088804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb99eb1-6386-46fe-9962-f3235cadab1f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:57.090360 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb99eb1-6386-46fe-9962-f3235cadab1f None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-3eb99eb1-6386-46fe-9962-f3235cadab1f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:57.105415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:57.106429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:57.107321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:57.107321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:57.107321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:57.107321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 266/570] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:57.107321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb99eb1-6386-46fe-9962-f3235cadab1f None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.124783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc5360b-cdbc-4574-bee3-22ef15ac3f94 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:57.161875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56dd415-d66d-4c49-97b8-3d79b4fac0a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:57.162538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56dd415-d66d-4c49-97b8-3d79b4fac0a4 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:57.162819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56dd415-d66d-4c49-97b8-3d79b4fac0a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:57.163222 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56dd415-d66d-4c49-97b8-3d79b4fac0a4 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:57.163871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56dd415-d66d-4c49-97b8-3d79b4fac0a4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:57.164173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56dd415-d66d-4c49-97b8-3d79b4fac0a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:57.165542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56dd415-d66d-4c49-97b8-3d79b4fac0a4 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:57.179402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 305/571] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:26:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:57.180091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56dd415-d66d-4c49-97b8-3d79b4fac0a4 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.195565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:57.217380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb99eb1-6386-46fe-9962-f3235cadab1f None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:57.254299 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2079a90e-4303-4e57-ae7b-5844390e026b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:57.254934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2079a90e-4303-4e57-ae7b-5844390e026b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:57.255213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2079a90e-4303-4e57-ae7b-5844390e026b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:57.255488 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2079a90e-4303-4e57-ae7b-5844390e026b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:57.266799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2079a90e-4303-4e57-ae7b-5844390e026b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:57.268236 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2079a90e-4303-4e57-ae7b-5844390e026b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:57.283039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56dd415-d66d-4c49-97b8-3d79b4fac0a4 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-2079a90e-4303-4e57-ae7b-5844390e026b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:57.283208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:57.284917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:57.286101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:57.286101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:57.286101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:57.286101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 267/572] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:57.286101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2079a90e-4303-4e57-ae7b-5844390e026b None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.318664 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:57.323812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d1e1b3-4610-4a82-a1d4-a0b00408adc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:57.324443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d1e1b3-4610-4a82-a1d4-a0b00408adc3 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:57.324721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d1e1b3-4610-4a82-a1d4-a0b00408adc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:57.325002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d1e1b3-4610-4a82-a1d4-a0b00408adc3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:57.332247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d1e1b3-4610-4a82-a1d4-a0b00408adc3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:57.333690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d1e1b3-4610-4a82-a1d4-a0b00408adc3 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-01d1e1b3-4610-4a82-a1d4-a0b00408adc3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:26:57.346140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:57.347423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:57.348314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:26:57.348314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:26:57.348314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:26:57.348314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:26:57.348314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:26:57.348314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:26:57.348314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:57.348314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:57.348314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 306/573] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:57.348314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d1e1b3-4610-4a82-a1d4-a0b00408adc3 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.391584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2079a90e-4303-4e57-ae7b-5844390e026b None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:57.424121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:57.428458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7092cd7d-dea9-4592-9443-04309b561f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:57.429073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7092cd7d-dea9-4592-9443-04309b561f3b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:57.429331 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7092cd7d-dea9-4592-9443-04309b561f3b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:57.429594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7092cd7d-dea9-4592-9443-04309b561f3b None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:57.430267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7092cd7d-dea9-4592-9443-04309b561f3b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:57.430567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7092cd7d-dea9-4592-9443-04309b561f3b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:57.431911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7092cd7d-dea9-4592-9443-04309b561f3b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:57.443942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 268/574] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:57.444723 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7092cd7d-dea9-4592-9443-04309b561f3b None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.448431 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d1e1b3-4610-4a82-a1d4-a0b00408adc3 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:57.482816 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb4c348-04e7-45d2-92e7-193a605c4e9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:57.483439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb4c348-04e7-45d2-92e7-193a605c4e9f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:57.483730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb4c348-04e7-45d2-92e7-193a605c4e9f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:57.483985 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb4c348-04e7-45d2-92e7-193a605c4e9f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:57.494077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb4c348-04e7-45d2-92e7-193a605c4e9f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:57.495550 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb4c348-04e7-45d2-92e7-193a605c4e9f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-3fb4c348-04e7-45d2-92e7-193a605c4e9f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:57.509824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:57.510803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:57.511710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:57.511710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:57.511710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:57.511710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 307/575] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:57.511710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb4c348-04e7-45d2-92e7-193a605c4e9f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.536982 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:57.548574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7092cd7d-dea9-4592-9443-04309b561f3b None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:57.584318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d47dec2-b920-4349-959c-95ed4fb96962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:57.584950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d47dec2-b920-4349-959c-95ed4fb96962 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:57.585228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d47dec2-b920-4349-959c-95ed4fb96962 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:57.585505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d47dec2-b920-4349-959c-95ed4fb96962 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:57.595913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d47dec2-b920-4349-959c-95ed4fb96962 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-9d47dec2-b920-4349-959c-95ed4fb96962 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:26:57.601756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:57.602905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 269/576] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:26:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:57.603931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d47dec2-b920-4349-959c-95ed4fb96962 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.615030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb4c348-04e7-45d2-92e7-193a605c4e9f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:57.639881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:57.649841 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03d8e9e-2395-43ca-967c-df45ac56296a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:57.650469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03d8e9e-2395-43ca-967c-df45ac56296a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:57.650746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03d8e9e-2395-43ca-967c-df45ac56296a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:57.651023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03d8e9e-2395-43ca-967c-df45ac56296a None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:57.651665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03d8e9e-2395-43ca-967c-df45ac56296a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:57.651962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03d8e9e-2395-43ca-967c-df45ac56296a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:57.653338 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03d8e9e-2395-43ca-967c-df45ac56296a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:57.665516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 308/577] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:57.666350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03d8e9e-2395-43ca-967c-df45ac56296a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.712702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d47dec2-b920-4349-959c-95ed4fb96962 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:57.745255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a2653-f780-4b44-a6b2-b96a2b115049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:57.745859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a2653-f780-4b44-a6b2-b96a2b115049 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:57.746137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a2653-f780-4b44-a6b2-b96a2b115049 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:57.746439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a2653-f780-4b44-a6b2-b96a2b115049 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:57.747127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a2653-f780-4b44-a6b2-b96a2b115049 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:57.747431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a2653-f780-4b44-a6b2-b96a2b115049 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:57.748851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a2653-f780-4b44-a6b2-b96a2b115049 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:57.760980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 270/578] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:26:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:57.761773 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a2653-f780-4b44-a6b2-b96a2b115049 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.763294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03d8e9e-2395-43ca-967c-df45ac56296a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:57.772507 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:57.798029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f30001-40ac-4b21-a6e4-c72f121ed10c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:57.798680 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f30001-40ac-4b21-a6e4-c72f121ed10c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:57.798957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f30001-40ac-4b21-a6e4-c72f121ed10c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:57.799224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f30001-40ac-4b21-a6e4-c72f121ed10c None admin] PATH_INFO: `/v3/users/glance` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:57.806371 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f30001-40ac-4b21-a6e4-c72f121ed10c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:57.807821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f30001-40ac-4b21-a6e4-c72f121ed10c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-d9f30001-40ac-4b21-a6e4-c72f121ed10c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:26:57.819025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:57.820035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:57.820987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:57.820987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:57.820987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:26:57.820987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:26:57.820987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:26:57.820987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:26:57.820987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:26:57.820987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:26:57.820987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 24 03:26:57.820987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:57.820987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 309/579] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:57 2022] GET /identity/v3/users/glance => generated 84 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:57.821577 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f30001-40ac-4b21-a6e4-c72f121ed10c None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.866058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a2653-f780-4b44-a6b2-b96a2b115049 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:57.869804 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:57.900340 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c601bd9e-d443-4346-9a2b-43d6c7f0ff3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:57.900961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c601bd9e-d443-4346-9a2b-43d6c7f0ff3d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:57.901239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c601bd9e-d443-4346-9a2b-43d6c7f0ff3d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:57.901509 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c601bd9e-d443-4346-9a2b-43d6c7f0ff3d None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:57.902143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c601bd9e-d443-4346-9a2b-43d6c7f0ff3d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:57.902524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c601bd9e-d443-4346-9a2b-43d6c7f0ff3d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:57.903881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c601bd9e-d443-4346-9a2b-43d6c7f0ff3d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:57.915147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 271/580] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:57.915817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c601bd9e-d443-4346-9a2b-43d6c7f0ff3d None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.923067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f30001-40ac-4b21-a6e4-c72f121ed10c None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:57.959217 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9316b3b-49be-4c3a-bbd0-3c890ed15b07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:57.959821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9316b3b-49be-4c3a-bbd0-3c890ed15b07 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:57.960104 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9316b3b-49be-4c3a-bbd0-3c890ed15b07 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:57.960378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9316b3b-49be-4c3a-bbd0-3c890ed15b07 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:57.961040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9316b3b-49be-4c3a-bbd0-3c890ed15b07 None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:57.961346 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9316b3b-49be-4c3a-bbd0-3c890ed15b07 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:57.962829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9316b3b-49be-4c3a-bbd0-3c890ed15b07 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:57.970662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 310/581] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:26:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:57.971419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9316b3b-49be-4c3a-bbd0-3c890ed15b07 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:57.985637 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:58.020618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c601bd9e-d443-4346-9a2b-43d6c7f0ff3d None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:58.054968 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d237e159-fdc9-44d6-b8d4-c52f5b5b2408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:58.055595 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d237e159-fdc9-44d6-b8d4-c52f5b5b2408 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:58.055926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d237e159-fdc9-44d6-b8d4-c52f5b5b2408 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:58.056140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d237e159-fdc9-44d6-b8d4-c52f5b5b2408 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:58.062857 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d237e159-fdc9-44d6-b8d4-c52f5b5b2408 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:58.064294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d237e159-fdc9-44d6-b8d4-c52f5b5b2408 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:58.071372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9316b3b-49be-4c3a-bbd0-3c890ed15b07 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-d237e159-fdc9-44d6-b8d4-c52f5b5b2408 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:26:58.075382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:58.076407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:58.077351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:26:58.077351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:26:58.077351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:26:58.077351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:26:58.077351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:26:58.077351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:26:58.077351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:58.077351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:58.077351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 272/582] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:58.077351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d237e159-fdc9-44d6-b8d4-c52f5b5b2408 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:58.081648 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:58.105179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41759f0-f080-404b-80b2-e2441c70eaed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:58.105826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41759f0-f080-404b-80b2-e2441c70eaed None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:58.106102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41759f0-f080-404b-80b2-e2441c70eaed None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:58.106410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41759f0-f080-404b-80b2-e2441c70eaed None admin] PATH_INFO: `/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/e5896018712d4b82a781365e478ee855/roles/94618a16f2e6449ba1cd5fbbd7c19f56` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:58.120883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41759f0-f080-404b-80b2-e2441c70eaed None admin] RBAC: Authorizing `identity:create_grant(project_id=5c653d5fa4304f298a295ce635efd5e6, user_id=e5896018712d4b82a781365e478ee855, role_id=94618a16f2e6449ba1cd5fbbd7c19f56)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:58.122342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41759f0-f080-404b-80b2-e2441c70eaed None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:58.178554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d237e159-fdc9-44d6-b8d4-c52f5b5b2408 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:58.189689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 311/583] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:26:57 2022] PUT /identity/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/e5896018712d4b82a781365e478ee855/roles/94618a16f2e6449ba1cd5fbbd7c19f56 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:58.190579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41759f0-f080-404b-80b2-e2441c70eaed None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:58.212715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:58.224391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b40d72d-b39e-4a92-97d4-2b4738f1f322 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:58.224945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b40d72d-b39e-4a92-97d4-2b4738f1f322 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:58.225228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b40d72d-b39e-4a92-97d4-2b4738f1f322 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:58.225503 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b40d72d-b39e-4a92-97d4-2b4738f1f322 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:58.226288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b40d72d-b39e-4a92-97d4-2b4738f1f322 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:58.226602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b40d72d-b39e-4a92-97d4-2b4738f1f322 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:58.228098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b40d72d-b39e-4a92-97d4-2b4738f1f322 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:58.253709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 273/584] 10.222.0.65 () {56 vars in 1137 bytes} [Wed Aug 24 03:26:58 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:58.253709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b40d72d-b39e-4a92-97d4-2b4738f1f322 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:58.308648 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41759f0-f080-404b-80b2-e2441c70eaed None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:58.361536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cba938-98f3-4b23-a7a6-4f8f070d8d65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:58.362247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cba938-98f3-4b23-a7a6-4f8f070d8d65 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:58.362588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cba938-98f3-4b23-a7a6-4f8f070d8d65 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:58.363000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cba938-98f3-4b23-a7a6-4f8f070d8d65 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:58.370262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cba938-98f3-4b23-a7a6-4f8f070d8d65 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:58.371701 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cba938-98f3-4b23-a7a6-4f8f070d8d65 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-40cba938-98f3-4b23-a7a6-4f8f070d8d65 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:26:58.385020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:58.385986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:58.386972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:58.386972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:58.386972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:26:58.386972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:26:58.386972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:26:58.386972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:26:58.386972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:26:58.386972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:26:58.386972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 24 03:26:58.386972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:58.386972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 312/585] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:26:58 2022] GET /identity/v3/users/neutron => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:58.390495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cba938-98f3-4b23-a7a6-4f8f070d8d65 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:58.401636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b40d72d-b39e-4a92-97d4-2b4738f1f322 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:58.416888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:58.447227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bca743b-bb5a-4a59-9df9-0f9ac0b4d11d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:58.448144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bca743b-bb5a-4a59-9df9-0f9ac0b4d11d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:58.448424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bca743b-bb5a-4a59-9df9-0f9ac0b4d11d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:58.448695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bca743b-bb5a-4a59-9df9-0f9ac0b4d11d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:58.460220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bca743b-bb5a-4a59-9df9-0f9ac0b4d11d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:58.461669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bca743b-bb5a-4a59-9df9-0f9ac0b4d11d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-8bca743b-bb5a-4a59-9df9-0f9ac0b4d11d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:58.480277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:58.481234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:58.482127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:58.482127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:58.482127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:58.482127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 274/586] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:58.482127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bca743b-bb5a-4a59-9df9-0f9ac0b4d11d None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:58.500035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cba938-98f3-4b23-a7a6-4f8f070d8d65 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:58.527406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:58.537405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdd9a81-afaa-4fcd-8bac-70348e10a81a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:58.538016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdd9a81-afaa-4fcd-8bac-70348e10a81a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:58.538341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdd9a81-afaa-4fcd-8bac-70348e10a81a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:58.538615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdd9a81-afaa-4fcd-8bac-70348e10a81a None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:58.541155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdd9a81-afaa-4fcd-8bac-70348e10a81a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:58.541155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdd9a81-afaa-4fcd-8bac-70348e10a81a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:58.541155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdd9a81-afaa-4fcd-8bac-70348e10a81a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:58.554390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 313/587] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:26:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:58.555333 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdd9a81-afaa-4fcd-8bac-70348e10a81a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:58.596029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bca743b-bb5a-4a59-9df9-0f9ac0b4d11d None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:58.636341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3abc043-8f67-4766-ae57-7ed6a4ea0a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:58.636974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3abc043-8f67-4766-ae57-7ed6a4ea0a57 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:58.637251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3abc043-8f67-4766-ae57-7ed6a4ea0a57 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:58.637957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3abc043-8f67-4766-ae57-7ed6a4ea0a57 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:58.641019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:58.651114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3abc043-8f67-4766-ae57-7ed6a4ea0a57 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:58.652591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3abc043-8f67-4766-ae57-7ed6a4ea0a57 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:58.663942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdd9a81-afaa-4fcd-8bac-70348e10a81a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-c3abc043-8f67-4766-ae57-7ed6a4ea0a57 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:58.667318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:58.668251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:58.674004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:58.674004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:58.674004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:58.674004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 275/588] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:58.674004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3abc043-8f67-4766-ae57-7ed6a4ea0a57 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:58.702707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8057fac-dc9a-4415-b719-804b89f8db89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:58.703320 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8057fac-dc9a-4415-b719-804b89f8db89 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:58.703597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8057fac-dc9a-4415-b719-804b89f8db89 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:58.703867 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8057fac-dc9a-4415-b719-804b89f8db89 None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:58.704594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8057fac-dc9a-4415-b719-804b89f8db89 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:58.704896 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8057fac-dc9a-4415-b719-804b89f8db89 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:58.706480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8057fac-dc9a-4415-b719-804b89f8db89 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:58.729075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 314/589] 10.222.0.65 () {56 vars in 1139 bytes} [Wed Aug 24 03:26:58 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:58.729075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8057fac-dc9a-4415-b719-804b89f8db89 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:58.776464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:58.786925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3abc043-8f67-4766-ae57-7ed6a4ea0a57 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:58.822678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba65a3e8-9249-4da7-b318-06ae93a16fcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:58.823280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba65a3e8-9249-4da7-b318-06ae93a16fcf None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:58.823559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba65a3e8-9249-4da7-b318-06ae93a16fcf None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:58.823831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba65a3e8-9249-4da7-b318-06ae93a16fcf None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:58.824463 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba65a3e8-9249-4da7-b318-06ae93a16fcf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:58.824760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba65a3e8-9249-4da7-b318-06ae93a16fcf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:58.826115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba65a3e8-9249-4da7-b318-06ae93a16fcf None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:58.837630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8057fac-dc9a-4415-b719-804b89f8db89 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:58.839223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 276/590] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:58.844566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba65a3e8-9249-4da7-b318-06ae93a16fcf None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:58.874591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec7f440-b033-4a92-9f77-a8ffff1bf4bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:58.875193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec7f440-b033-4a92-9f77-a8ffff1bf4bf None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:58.875469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec7f440-b033-4a92-9f77-a8ffff1bf4bf None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:58.875763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec7f440-b033-4a92-9f77-a8ffff1bf4bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:58.876476 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec7f440-b033-4a92-9f77-a8ffff1bf4bf None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=4591bec74a4641af818b2d75b657115e) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:58.876783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec7f440-b033-4a92-9f77-a8ffff1bf4bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:58.878777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec7f440-b033-4a92-9f77-a8ffff1bf4bf None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:58.882461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:58.890184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 315/591] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:26:58 2022] GET /identity/v3/role_assignments?user.id=4591bec74a4641af818b2d75b657115e&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:58.890184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec7f440-b033-4a92-9f77-a8ffff1bf4bf None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:58.969761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba65a3e8-9249-4da7-b318-06ae93a16fcf None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:59.013672 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cdc8c2-b87f-4ec0-9897-3de99a703b70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:59.014679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cdc8c2-b87f-4ec0-9897-3de99a703b70 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:59.015139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cdc8c2-b87f-4ec0-9897-3de99a703b70 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:59.015587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cdc8c2-b87f-4ec0-9897-3de99a703b70 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:59.016521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cdc8c2-b87f-4ec0-9897-3de99a703b70 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:59.017147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cdc8c2-b87f-4ec0-9897-3de99a703b70 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:59.018887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cdc8c2-b87f-4ec0-9897-3de99a703b70 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:59.038274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 277/592] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:59.038274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cdc8c2-b87f-4ec0-9897-3de99a703b70 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:59.046704 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec7f440-b033-4a92-9f77-a8ffff1bf4bf None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:59.051843 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:59.086527 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768543f0-5441-4bbe-bb0d-7ffe1fa04056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:59.087405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768543f0-5441-4bbe-bb0d-7ffe1fa04056 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:59.087833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768543f0-5441-4bbe-bb0d-7ffe1fa04056 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:59.088249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768543f0-5441-4bbe-bb0d-7ffe1fa04056 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:59.100881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768543f0-5441-4bbe-bb0d-7ffe1fa04056 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:59.102629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768543f0-5441-4bbe-bb0d-7ffe1fa04056 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-768543f0-5441-4bbe-bb0d-7ffe1fa04056 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:59.119414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:59.120597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:59.121535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:59.121535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:59.121535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:59.123506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 316/593] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:59.124501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768543f0-5441-4bbe-bb0d-7ffe1fa04056 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:59.161406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cdc8c2-b87f-4ec0-9897-3de99a703b70 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:59.193290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:59.200715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2741d7-b64b-4ebb-a663-5665899ff0f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:59.201709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2741d7-b64b-4ebb-a663-5665899ff0f1 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:59.202137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2741d7-b64b-4ebb-a663-5665899ff0f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:59.202574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2741d7-b64b-4ebb-a663-5665899ff0f1 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:59.212618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2741d7-b64b-4ebb-a663-5665899ff0f1 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:59.214755 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2741d7-b64b-4ebb-a663-5665899ff0f1 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-dc2741d7-b64b-4ebb-a663-5665899ff0f1 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:26:59.231258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:59.232303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:59.233282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:59.233282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:59.233282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:26:59.233282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:26:59.233282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:26:59.233282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:26:59.233282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:26:59.233282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:26:59.233282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 03:26:59.233282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:59.238252 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 278/594] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:26:59 2022] GET /identity/v3/users/cinder => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:59.240242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2741d7-b64b-4ebb-a663-5665899ff0f1 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:59.256833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768543f0-5441-4bbe-bb0d-7ffe1fa04056 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:59.295881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f84ef45-32fd-44ad-a2cc-78ed8516e716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:59.296829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f84ef45-32fd-44ad-a2cc-78ed8516e716 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:59.297213 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f84ef45-32fd-44ad-a2cc-78ed8516e716 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:59.297602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f84ef45-32fd-44ad-a2cc-78ed8516e716 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:59.305853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f84ef45-32fd-44ad-a2cc-78ed8516e716 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:59.307622 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f84ef45-32fd-44ad-a2cc-78ed8516e716 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-6f84ef45-32fd-44ad-a2cc-78ed8516e716 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:26:59.320006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:59.321094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:59.322196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:26:59.322196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:26:59.322196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:26:59.322196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:26:59.322196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:26:59.322196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:26:59.322196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:59.322196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:59.322719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:59.324773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 317/595] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:59.325577 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f84ef45-32fd-44ad-a2cc-78ed8516e716 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:59.363226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2741d7-b64b-4ebb-a663-5665899ff0f1 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:59.399986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7118db5-f24c-4056-af7b-49597c3c36a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:59.400688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7118db5-f24c-4056-af7b-49597c3c36a5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:59.401042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7118db5-f24c-4056-af7b-49597c3c36a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:59.401386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7118db5-f24c-4056-af7b-49597c3c36a5 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:59.402229 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7118db5-f24c-4056-af7b-49597c3c36a5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:59.402606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7118db5-f24c-4056-af7b-49597c3c36a5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:59.404145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7118db5-f24c-4056-af7b-49597c3c36a5 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:59.428693 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 279/596] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:59.430172 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7118db5-f24c-4056-af7b-49597c3c36a5 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:59.436422 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f84ef45-32fd-44ad-a2cc-78ed8516e716 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:59.450879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:59.474455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b92ffa-3ff8-4624-a656-35dfde2776d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:59.475202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b92ffa-3ff8-4624-a656-35dfde2776d5 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:59.475559 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b92ffa-3ff8-4624-a656-35dfde2776d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:59.475981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b92ffa-3ff8-4624-a656-35dfde2776d5 None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:59.476866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b92ffa-3ff8-4624-a656-35dfde2776d5 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:59.477241 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b92ffa-3ff8-4624-a656-35dfde2776d5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:59.478942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b92ffa-3ff8-4624-a656-35dfde2776d5 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:59.499251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 318/597] 10.222.0.65 () {56 vars in 1137 bytes} [Wed Aug 24 03:26:59 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:59.500140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b92ffa-3ff8-4624-a656-35dfde2776d5 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:59.540464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7118db5-f24c-4056-af7b-49597c3c36a5 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:59.571479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:59.577578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb1e8fe-0dd2-429a-b806-0f062fe05441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:59.578607 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1e8fe-0dd2-429a-b806-0f062fe05441 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:59.579085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1e8fe-0dd2-429a-b806-0f062fe05441 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:59.579519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1e8fe-0dd2-429a-b806-0f062fe05441 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:59.584738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb1e8fe-0dd2-429a-b806-0f062fe05441 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:59.584738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb1e8fe-0dd2-429a-b806-0f062fe05441 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:59.584738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb1e8fe-0dd2-429a-b806-0f062fe05441 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:59.600838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 280/598] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:26:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:26:59.602598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb1e8fe-0dd2-429a-b806-0f062fe05441 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:59.637957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b92ffa-3ff8-4624-a656-35dfde2776d5 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:59.674840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee0d4d3-9e7c-4f1e-ad6d-12b8a4bc7141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:59.675856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee0d4d3-9e7c-4f1e-ad6d-12b8a4bc7141 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:59.676335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee0d4d3-9e7c-4f1e-ad6d-12b8a4bc7141 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:59.676803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee0d4d3-9e7c-4f1e-ad6d-12b8a4bc7141 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:59.683818 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee0d4d3-9e7c-4f1e-ad6d-12b8a4bc7141 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:59.685540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee0d4d3-9e7c-4f1e-ad6d-12b8a4bc7141 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-0ee0d4d3-9e7c-4f1e-ad6d-12b8a4bc7141 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:26:59.697795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:59.698907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:59.699802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:26:59.699802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:26:59.699802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:26:59.699802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:26:59.699802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:26:59.699802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:26:59.699802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:26:59.699802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:26:59.702117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 319/599] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:26:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:59.703203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee0d4d3-9e7c-4f1e-ad6d-12b8a4bc7141 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:59.721372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb1e8fe-0dd2-429a-b806-0f062fe05441 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:59.734459 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:59.766836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afb4c5f-5c90-4602-8d97-e26956e9bd82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:59.767473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afb4c5f-5c90-4602-8d97-e26956e9bd82 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:59.767748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afb4c5f-5c90-4602-8d97-e26956e9bd82 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:59.768020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afb4c5f-5c90-4602-8d97-e26956e9bd82 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:59.782316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afb4c5f-5c90-4602-8d97-e26956e9bd82 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:59.784777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afb4c5f-5c90-4602-8d97-e26956e9bd82 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-7afb4c5f-5c90-4602-8d97-e26956e9bd82 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:26:59.804205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:26:59.805188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:26:59.806107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:26:59.806107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:26:59.806107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:26:59.810924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 281/600] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:26:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:26:59.811918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afb4c5f-5c90-4602-8d97-e26956e9bd82 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:59.818532 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee0d4d3-9e7c-4f1e-ad6d-12b8a4bc7141 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:59.854452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d17ddf1-ab10-4256-85ab-a5b3ea77abb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:59.855168 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d17ddf1-ab10-4256-85ab-a5b3ea77abb1 None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:59.855446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d17ddf1-ab10-4256-85ab-a5b3ea77abb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:59.855715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d17ddf1-ab10-4256-85ab-a5b3ea77abb1 None admin] PATH_INFO: `/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/9f3b3c7ebc08496e970f8f25fd6395e4/roles/94618a16f2e6449ba1cd5fbbd7c19f56` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:59.871205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d17ddf1-ab10-4256-85ab-a5b3ea77abb1 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c653d5fa4304f298a295ce635efd5e6, user_id=9f3b3c7ebc08496e970f8f25fd6395e4, role_id=94618a16f2e6449ba1cd5fbbd7c19f56)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:26:59.872569 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:26:59.872681 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d17ddf1-ab10-4256-85ab-a5b3ea77abb1 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:26:59.946421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 320/601] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:26:59 2022] PUT /identity/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/9f3b3c7ebc08496e970f8f25fd6395e4/roles/94618a16f2e6449ba1cd5fbbd7c19f56 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:26:59.947699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d17ddf1-ab10-4256-85ab-a5b3ea77abb1 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:26:59.948206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afb4c5f-5c90-4602-8d97-e26956e9bd82 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:26:59.995679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d10af9-d2e4-4abd-84ea-97e364a41016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:26:59.996533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d10af9-d2e4-4abd-84ea-97e364a41016 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:26:59.996967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d10af9-d2e4-4abd-84ea-97e364a41016 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:26:59.997390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d10af9-d2e4-4abd-84ea-97e364a41016 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:26:59.998989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d10af9-d2e4-4abd-84ea-97e364a41016 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:26:59.999513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d10af9-d2e4-4abd-84ea-97e364a41016 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:00.001464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d10af9-d2e4-4abd-84ea-97e364a41016 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:00.019096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 282/602] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:26:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:00.025722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d10af9-d2e4-4abd-84ea-97e364a41016 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:00.056491 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:00.078948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d17ddf1-ab10-4256-85ab-a5b3ea77abb1 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:00.118069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149f4911-15bd-4b1c-836d-b23eed2f395f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:00.118646 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149f4911-15bd-4b1c-836d-b23eed2f395f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:00.118926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149f4911-15bd-4b1c-836d-b23eed2f395f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:00.119197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149f4911-15bd-4b1c-836d-b23eed2f395f None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:00.119968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149f4911-15bd-4b1c-836d-b23eed2f395f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:00.120268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149f4911-15bd-4b1c-836d-b23eed2f395f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:00.121905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149f4911-15bd-4b1c-836d-b23eed2f395f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:00.153497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 321/603] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:26:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:00.158570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d10af9-d2e4-4abd-84ea-97e364a41016 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:00.161038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:00.165534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56df694-5a0f-46b0-867f-fe800523bc62 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:00.165820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56df694-5a0f-46b0-867f-fe800523bc62 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:00.166098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56df694-5a0f-46b0-867f-fe800523bc62 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:00.167702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 322/604] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:00.171288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56df694-5a0f-46b0-867f-fe800523bc62 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:00.204145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ac7212-f12a-452f-be76-51b24022e347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:00.205763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ac7212-f12a-452f-be76-51b24022e347 None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:00.205763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ac7212-f12a-452f-be76-51b24022e347 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:00.205763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ac7212-f12a-452f-be76-51b24022e347 None admin] PATH_INFO: `/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/64f23f0c8abc406b9700361723623387/roles/94618a16f2e6449ba1cd5fbbd7c19f56` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:00.223807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ac7212-f12a-452f-be76-51b24022e347 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c653d5fa4304f298a295ce635efd5e6, user_id=64f23f0c8abc406b9700361723623387, role_id=94618a16f2e6449ba1cd5fbbd7c19f56)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:00.225462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ac7212-f12a-452f-be76-51b24022e347 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:00.294469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56df694-5a0f-46b0-867f-fe800523bc62 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:00.317239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 283/605] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:27:00 2022] PUT /identity/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/64f23f0c8abc406b9700361723623387/roles/94618a16f2e6449ba1cd5fbbd7c19f56 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:27:00.320088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484f0072-ff23-403a-a91c-099e2a839bc4 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:00.320774 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484f0072-ff23-403a-a91c-099e2a839bc4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:00.321352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484f0072-ff23-403a-a91c-099e2a839bc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:00.348086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-484f0072-ff23-403a-a91c-099e2a839bc4 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:00.351064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-484f0072-ff23-403a-a91c-099e2a839bc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:00.365055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb53661-a2d8-4cf5-bda1-e1e529340f44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:00.365326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb53661-a2d8-4cf5-bda1-e1e529340f44 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:00.365502 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb53661-a2d8-4cf5-bda1-e1e529340f44 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:00.365775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb53661-a2d8-4cf5-bda1-e1e529340f44 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:00.388141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb53661-a2d8-4cf5-bda1-e1e529340f44 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:00.390733 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb53661-a2d8-4cf5-bda1-e1e529340f44 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:00.404007 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-8eb53661-a2d8-4cf5-bda1-e1e529340f44 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:00.423412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:00.424589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:00.425837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:00.425837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:00.425837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:00.425837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:00.425837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:00.425837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:00.425837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:00.425837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:00.429819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 323/606] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:00.431269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb53661-a2d8-4cf5-bda1-e1e529340f44 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:00.471442 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 284/607] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:27:00.571993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb53661-a2d8-4cf5-bda1-e1e529340f44 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:00.608256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e36947b-c0f5-4682-b730-615319e4da9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:00.609182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e36947b-c0f5-4682-b730-615319e4da9d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:00.609624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e36947b-c0f5-4682-b730-615319e4da9d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:00.610043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e36947b-c0f5-4682-b730-615319e4da9d None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:00.610895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e36947b-c0f5-4682-b730-615319e4da9d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:00.611344 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e36947b-c0f5-4682-b730-615319e4da9d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:00.613082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e36947b-c0f5-4682-b730-615319e4da9d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:00.627299 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 324/608] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:00.634138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484f0072-ff23-403a-a91c-099e2a839bc4 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:00.747216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:00.770843 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484f0072-ff23-403a-a91c-099e2a839bc4 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:00.822969 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454e2871-632b-4443-823d-4f79529b8b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:00.823636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454e2871-632b-4443-823d-4f79529b8b5a None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:00.823914 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454e2871-632b-4443-823d-4f79529b8b5a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:00.824187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454e2871-632b-4443-823d-4f79529b8b5a None admin] PATH_INFO: `/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/dbfbfd64573141f88040692119ac5a1f/roles/94618a16f2e6449ba1cd5fbbd7c19f56` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:00.837135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e36947b-c0f5-4682-b730-615319e4da9d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:00.854621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454e2871-632b-4443-823d-4f79529b8b5a None admin] RBAC: Authorizing `identity:create_grant(project_id=5c653d5fa4304f298a295ce635efd5e6, user_id=dbfbfd64573141f88040692119ac5a1f, role_id=94618a16f2e6449ba1cd5fbbd7c19f56)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:00.859926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454e2871-632b-4443-823d-4f79529b8b5a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:00.914124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:00.933813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 285/609] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:27:00 2022] PUT /identity/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/dbfbfd64573141f88040692119ac5a1f/roles/94618a16f2e6449ba1cd5fbbd7c19f56 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:27:00.971107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e36947b-c0f5-4682-b730-615319e4da9d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:01.011837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c81dbe9-65b1-4dd6-a1c2-2e167beaddb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:01.012813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c81dbe9-65b1-4dd6-a1c2-2e167beaddb8 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:01.013315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c81dbe9-65b1-4dd6-a1c2-2e167beaddb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:01.013806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c81dbe9-65b1-4dd6-a1c2-2e167beaddb8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:01.026379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c81dbe9-65b1-4dd6-a1c2-2e167beaddb8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-0c81dbe9-65b1-4dd6-a1c2-2e167beaddb8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:01.031834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:01.032881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:01.033744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:01.033744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:01.033744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:01.033744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:01.033744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:01.033744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:01.033744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:01.033744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:01.033744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:01.033744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:01.033744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:01.033744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:01.037177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 325/610] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:27:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:01.043295 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454e2871-632b-4443-823d-4f79529b8b5a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:01.148539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc484f14-1d54-4f10-a2ca-7138ac8f732f None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:01.148886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc484f14-1d54-4f10-a2ca-7138ac8f732f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:01.149167 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc484f14-1d54-4f10-a2ca-7138ac8f732f None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:01.150697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 326/611] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:01.163572 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3303b75-aa31-402d-9c35-c9aefc3db94f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:01.164198 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3303b75-aa31-402d-9c35-c9aefc3db94f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:01.164662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3303b75-aa31-402d-9c35-c9aefc3db94f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:01.171259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454e2871-632b-4443-823d-4f79529b8b5a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:01.199457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e3303b75-aa31-402d-9c35-c9aefc3db94f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:01.200745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e3303b75-aa31-402d-9c35-c9aefc3db94f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:01.223385 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:01.228419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0d8bbb-85f6-4703-b924-0d47aee8f8a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:01.229037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0d8bbb-85f6-4703-b924-0d47aee8f8a3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:01.229316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0d8bbb-85f6-4703-b924-0d47aee8f8a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:01.229590 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0d8bbb-85f6-4703-b924-0d47aee8f8a3 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:01.230302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0d8bbb-85f6-4703-b924-0d47aee8f8a3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:01.230607 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0d8bbb-85f6-4703-b924-0d47aee8f8a3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:01.231981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0d8bbb-85f6-4703-b924-0d47aee8f8a3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:01.241917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 286/612] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:27:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:01.248426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0d8bbb-85f6-4703-b924-0d47aee8f8a3 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:01.298803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 327/613] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:27:01.374674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0d8bbb-85f6-4703-b924-0d47aee8f8a3 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:01.417852 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3527d286-e7c3-46ec-bc5a-c7bde1be8984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:01.418703 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3527d286-e7c3-46ec-bc5a-c7bde1be8984 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:01.419121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3527d286-e7c3-46ec-bc5a-c7bde1be8984 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:01.419511 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3527d286-e7c3-46ec-bc5a-c7bde1be8984 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:01.438638 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3527d286-e7c3-46ec-bc5a-c7bde1be8984 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:01.440581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3527d286-e7c3-46ec-bc5a-c7bde1be8984 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-3527d286-e7c3-46ec-bc5a-c7bde1be8984 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:01.460500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:01.461683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:01.462806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:01.462806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:01.462806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:01.464536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 287/614] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:01.470088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3303b75-aa31-402d-9c35-c9aefc3db94f None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:01.545462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:01.610348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3303b75-aa31-402d-9c35-c9aefc3db94f None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:01.648632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3527d286-e7c3-46ec-bc5a-c7bde1be8984 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:01.671347 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b948c4f9-55f7-46b3-a16c-36a9ed916a13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:01.672320 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b948c4f9-55f7-46b3-a16c-36a9ed916a13 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:01.672839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b948c4f9-55f7-46b3-a16c-36a9ed916a13 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:01.673307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b948c4f9-55f7-46b3-a16c-36a9ed916a13 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:01.674182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b948c4f9-55f7-46b3-a16c-36a9ed916a13 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:01.675172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b948c4f9-55f7-46b3-a16c-36a9ed916a13 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:01.677726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b948c4f9-55f7-46b3-a16c-36a9ed916a13 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:01.705210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 328/615] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:01.710601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b948c4f9-55f7-46b3-a16c-36a9ed916a13 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:01.722412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:01.802981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3527d286-e7c3-46ec-bc5a-c7bde1be8984 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:01.870801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b669af4-9b35-406a-928a-99fc70c282eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:01.871352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b669af4-9b35-406a-928a-99fc70c282eb None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:01.871628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b669af4-9b35-406a-928a-99fc70c282eb None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:01.871897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b669af4-9b35-406a-928a-99fc70c282eb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:01.876260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b948c4f9-55f7-46b3-a16c-36a9ed916a13 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:01.897358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b669af4-9b35-406a-928a-99fc70c282eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:01.902313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-5b669af4-9b35-406a-928a-99fc70c282eb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:01.908355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:01.910002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:01.917294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:01.917294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:01.917294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:01.917294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:01.917294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:01.917294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:01.917294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:01.917294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:01.917294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:01.917294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:01.917294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:01.917294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:01.919060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 288/616] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:27:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:01.920035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b669af4-9b35-406a-928a-99fc70c282eb None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:01.954863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61eaf10-5d57-47d0-aad9-9a3f130cf852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:01.955880 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61eaf10-5d57-47d0-aad9-9a3f130cf852 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:01.956479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61eaf10-5d57-47d0-aad9-9a3f130cf852 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:01.957175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61eaf10-5d57-47d0-aad9-9a3f130cf852 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:01.968677 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61eaf10-5d57-47d0-aad9-9a3f130cf852 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:01.970461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61eaf10-5d57-47d0-aad9-9a3f130cf852 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-d61eaf10-5d57-47d0-aad9-9a3f130cf852 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:01.998610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:01.999627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:02.000599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:02.000599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:02.000599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:02.000599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:02.000599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:02.000599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:02.000599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:02.000599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:02.000599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:02.000599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:02.007108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 329/617] 10.222.0.65 () {56 vars in 1093 bytes} [Wed Aug 24 03:27:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:02.013923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61eaf10-5d57-47d0-aad9-9a3f130cf852 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:02.128582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b669af4-9b35-406a-928a-99fc70c282eb None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:02.153317 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:02.181085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61eaf10-5d57-47d0-aad9-9a3f130cf852 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:02.199802 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe1eea3-2084-435a-9c18-4c96309b5b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:02.200424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe1eea3-2084-435a-9c18-4c96309b5b15 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:02.200704 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe1eea3-2084-435a-9c18-4c96309b5b15 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:02.201038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe1eea3-2084-435a-9c18-4c96309b5b15 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:02.201707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe1eea3-2084-435a-9c18-4c96309b5b15 None admin] RBAC: Adding query filter params (name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:02.202009 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe1eea3-2084-435a-9c18-4c96309b5b15 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:02.203428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe1eea3-2084-435a-9c18-4c96309b5b15 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:02.213358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 289/618] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:27:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:02.221008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe1eea3-2084-435a-9c18-4c96309b5b15 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:02.241474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b338510-2c55-45d3-99e8-7202eac71c1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:02.242086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b338510-2c55-45d3-99e8-7202eac71c1f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:02.242390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b338510-2c55-45d3-99e8-7202eac71c1f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:02.242661 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b338510-2c55-45d3-99e8-7202eac71c1f None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:02.243397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b338510-2c55-45d3-99e8-7202eac71c1f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:02.243702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b338510-2c55-45d3-99e8-7202eac71c1f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:02.245154 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b338510-2c55-45d3-99e8-7202eac71c1f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:02.278904 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 330/619] 10.222.0.65 () {56 vars in 1133 bytes} [Wed Aug 24 03:27:02 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:02.287168 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b338510-2c55-45d3-99e8-7202eac71c1f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:02.341352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:02.374145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe1eea3-2084-435a-9c18-4c96309b5b15 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:02.415457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6453106-7f54-4dbc-89da-8649fa38a41d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:02.416041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6453106-7f54-4dbc-89da-8649fa38a41d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:02.416312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6453106-7f54-4dbc-89da-8649fa38a41d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:02.416578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6453106-7f54-4dbc-89da-8649fa38a41d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:02.443479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6453106-7f54-4dbc-89da-8649fa38a41d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:02.444919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6453106-7f54-4dbc-89da-8649fa38a41d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-e6453106-7f54-4dbc-89da-8649fa38a41d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:02.471702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:02.472753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:02.473661 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:02.473661 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:02.473661 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:02.473856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 290/620] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:02.475008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0096d0-16b3-41d5-8f5e-5fa4da814daa None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:02.475114 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b338510-2c55-45d3-99e8-7202eac71c1f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:02.475289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0096d0-16b3-41d5-8f5e-5fa4da814daa None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:02.476771 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0096d0-16b3-41d5-8f5e-5fa4da814daa None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:02.478858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 291/621] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:02.481593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0096d0-16b3-41d5-8f5e-5fa4da814daa None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:02.517133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:02.521583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ea17f5-5b79-43c2-92cb-d23696a007d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:02.522219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea17f5-5b79-43c2-92cb-d23696a007d3 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:02.522535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea17f5-5b79-43c2-92cb-d23696a007d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:02.522821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea17f5-5b79-43c2-92cb-d23696a007d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:02.593954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ea17f5-5b79-43c2-92cb-d23696a007d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:02.595525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ea17f5-5b79-43c2-92cb-d23696a007d3 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:02.603574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0096d0-16b3-41d5-8f5e-5fa4da814daa None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:02.625106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:02.640049 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dffd837-8cea-4b1b-b97d-92d77ae1abcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:02.640734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dffd837-8cea-4b1b-b97d-92d77ae1abcd None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:02.641010 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dffd837-8cea-4b1b-b97d-92d77ae1abcd None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:02.641275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dffd837-8cea-4b1b-b97d-92d77ae1abcd None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:02.641909 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dffd837-8cea-4b1b-b97d-92d77ae1abcd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:02.642226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dffd837-8cea-4b1b-b97d-92d77ae1abcd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:02.643754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dffd837-8cea-4b1b-b97d-92d77ae1abcd None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:02.655429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 292/622] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:02.656624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa8e9a4-9580-47cc-91ce-ef8029b506b5 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:02.656911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa8e9a4-9580-47cc-91ce-ef8029b506b5 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:02.657183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa8e9a4-9580-47cc-91ce-ef8029b506b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:02.675029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1fa8e9a4-9580-47cc-91ce-ef8029b506b5 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:02.676450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1fa8e9a4-9580-47cc-91ce-ef8029b506b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:02.712358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 331/623] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:27:02.713680 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be3feca-8f1d-4a2e-925b-6e23f24d9a88 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:02.713978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be3feca-8f1d-4a2e-925b-6e23f24d9a88 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:02.714276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be3feca-8f1d-4a2e-925b-6e23f24d9a88 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:02.715571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 332/624] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:02.716396 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be3feca-8f1d-4a2e-925b-6e23f24d9a88 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:02.762505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 293/625] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:27:02.764086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa8e9a4-9580-47cc-91ce-ef8029b506b5 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:02.781226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:02.838392 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be3feca-8f1d-4a2e-925b-6e23f24d9a88 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:02.876503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:02.883832 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa8e9a4-9580-47cc-91ce-ef8029b506b5 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:02.889253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427755f5-b524-46f4-9301-9d407f055f3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:02.890284 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427755f5-b524-46f4-9301-9d407f055f3d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:02.890758 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427755f5-b524-46f4-9301-9d407f055f3d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:02.891239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427755f5-b524-46f4-9301-9d407f055f3d None admin] PATH_INFO: `/v3/users/swift` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:02.902352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427755f5-b524-46f4-9301-9d407f055f3d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:02.904446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427755f5-b524-46f4-9301-9d407f055f3d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-427755f5-b524-46f4-9301-9d407f055f3d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:02.922780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:02.923967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 333/626] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:02 2022] GET /identity/v3/users/swift => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20ae378-5fec-476b-b20f-0b409394c678 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:02.930061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20ae378-5fec-476b-b20f-0b409394c678 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:02.930952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20ae378-5fec-476b-b20f-0b409394c678 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:02.934380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7200b861-2ce0-4b5f-af07-9096b3c02c77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:02.935310 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7200b861-2ce0-4b5f-af07-9096b3c02c77 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:02.935803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7200b861-2ce0-4b5f-af07-9096b3c02c77 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:02.936280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7200b861-2ce0-4b5f-af07-9096b3c02c77 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:02.947735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e20ae378-5fec-476b-b20f-0b409394c678 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:02.948381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7200b861-2ce0-4b5f-af07-9096b3c02c77 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:02.948947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e20ae378-5fec-476b-b20f-0b409394c678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:02.950415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7200b861-2ce0-4b5f-af07-9096b3c02c77 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-7200b861-2ce0-4b5f-af07-9096b3c02c77 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:02.967566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:02.968581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:02.969474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:02.969474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:02.969474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:02.971676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 294/627] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:02.972898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7200b861-2ce0-4b5f-af07-9096b3c02c77 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:03.038432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 334/628] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:27:03.039648 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20ae378-5fec-476b-b20f-0b409394c678 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:03.045485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:03.093005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7200b861-2ce0-4b5f-af07-9096b3c02c77 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:03.136875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc4495d-ef56-419e-9635-2a17eef6de27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:03.138053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc4495d-ef56-419e-9635-2a17eef6de27 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:03.138435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc4495d-ef56-419e-9635-2a17eef6de27 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:03.138781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc4495d-ef56-419e-9635-2a17eef6de27 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:03.139975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc4495d-ef56-419e-9635-2a17eef6de27 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:03.140544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc4495d-ef56-419e-9635-2a17eef6de27 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:03.142258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc4495d-ef56-419e-9635-2a17eef6de27 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:03.165062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 295/629] 10.222.0.65 () {56 vars in 1135 bytes} [Wed Aug 24 03:27:02 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:03.165308 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20ae378-5fec-476b-b20f-0b409394c678 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:03.165892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc4495d-ef56-419e-9635-2a17eef6de27 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:03.188109 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:03.209798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72aab90-58f5-44ed-b748-f8d334168931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:03.212922 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72aab90-58f5-44ed-b748-f8d334168931 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:03.212922 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72aab90-58f5-44ed-b748-f8d334168931 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:03.212922 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72aab90-58f5-44ed-b748-f8d334168931 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:03.212922 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72aab90-58f5-44ed-b748-f8d334168931 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:03.212922 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72aab90-58f5-44ed-b748-f8d334168931 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:03.213986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72aab90-58f5-44ed-b748-f8d334168931 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:03.233753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 335/630] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:03.234782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72aab90-58f5-44ed-b748-f8d334168931 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:03.288017 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc4495d-ef56-419e-9635-2a17eef6de27 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:03.314604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:03.325471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a5a0dc-c309-42ac-9f94-189b9edcfa15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:03.326402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a5a0dc-c309-42ac-9f94-189b9edcfa15 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:03.326834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a5a0dc-c309-42ac-9f94-189b9edcfa15 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:03.327260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a5a0dc-c309-42ac-9f94-189b9edcfa15 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:03.337714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a5a0dc-c309-42ac-9f94-189b9edcfa15 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-32a5a0dc-c309-42ac-9f94-189b9edcfa15 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:03.342660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:03.343588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:03.344435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:03.344435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:03.344435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:03.344435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:03.344435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:03.344435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:03.344435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:03.344435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:03.344435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:03.344435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:03.344435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:03.344435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:03.347260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 296/631] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:27:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:03.348400 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a5a0dc-c309-42ac-9f94-189b9edcfa15 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:03.358740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72aab90-58f5-44ed-b748-f8d334168931 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:03.392426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a87a25-e448-498a-adc4-0a1795e1aa24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:03.393049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a87a25-e448-498a-adc4-0a1795e1aa24 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:03.393344 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a87a25-e448-498a-adc4-0a1795e1aa24 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:03.393639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a87a25-e448-498a-adc4-0a1795e1aa24 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:03.403767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a87a25-e448-498a-adc4-0a1795e1aa24 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:03.405192 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a87a25-e448-498a-adc4-0a1795e1aa24 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-29a87a25-e448-498a-adc4-0a1795e1aa24 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:03.420197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:03.421136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:03.422032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:03.422032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:03.422032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:03.422032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 336/632] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:03.422032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a87a25-e448-498a-adc4-0a1795e1aa24 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:03.451954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:03.454783 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a5a0dc-c309-42ac-9f94-189b9edcfa15 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:03.489001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74504ce7-89fa-4389-a7ae-74688a8010cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:03.489616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74504ce7-89fa-4389-a7ae-74688a8010cc None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:03.489895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74504ce7-89fa-4389-a7ae-74688a8010cc None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:03.490175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74504ce7-89fa-4389-a7ae-74688a8010cc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:03.496692 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74504ce7-89fa-4389-a7ae-74688a8010cc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:03.498131 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74504ce7-89fa-4389-a7ae-74688a8010cc None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-74504ce7-89fa-4389-a7ae-74688a8010cc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:03.509076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:03.509991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:03.510912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:03.510912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:03.510912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:03.510912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:03.510912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:03.510912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:03.510912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:03.510912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:03.510912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 297/633] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:03.510912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74504ce7-89fa-4389-a7ae-74688a8010cc None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:03.523590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a87a25-e448-498a-adc4-0a1795e1aa24 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:03.555253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:03.557036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374bb0d3-15c3-4a16-b981-91e63914aeec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:03.557662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374bb0d3-15c3-4a16-b981-91e63914aeec None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:03.557943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374bb0d3-15c3-4a16-b981-91e63914aeec None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:03.558246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374bb0d3-15c3-4a16-b981-91e63914aeec None admin] PATH_INFO: `/v3/roles/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:03.569391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374bb0d3-15c3-4a16-b981-91e63914aeec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-374bb0d3-15c3-4a16-b981-91e63914aeec None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:03.574587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:03.575671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 337/634] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:27:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:03.577207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12550d3c-dea5-4d09-9664-3391cd5d6b7c None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:03.578081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12550d3c-dea5-4d09-9664-3391cd5d6b7c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:03.578081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12550d3c-dea5-4d09-9664-3391cd5d6b7c None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:03.578777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 338/635] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:03 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:03.579350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12550d3c-dea5-4d09-9664-3391cd5d6b7c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:03.621007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74504ce7-89fa-4389-a7ae-74688a8010cc None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:03.655794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f34e08-188d-437b-a850-ee5b9caedda8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:03.656377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f34e08-188d-437b-a850-ee5b9caedda8 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:03.656655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f34e08-188d-437b-a850-ee5b9caedda8 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:03.656924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f34e08-188d-437b-a850-ee5b9caedda8 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:03.657581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f34e08-188d-437b-a850-ee5b9caedda8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:03.657877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f34e08-188d-437b-a850-ee5b9caedda8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:03.659323 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f34e08-188d-437b-a850-ee5b9caedda8 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:03.667213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 298/636] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:27:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:03.668004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f34e08-188d-437b-a850-ee5b9caedda8 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:03.669360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:03.681024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12550d3c-dea5-4d09-9664-3391cd5d6b7c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:03.714432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cc6956-dbbf-4485-9826-5d378e75dd92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:03.714963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cc6956-dbbf-4485-9826-5d378e75dd92 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:03.715240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cc6956-dbbf-4485-9826-5d378e75dd92 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:03.715511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cc6956-dbbf-4485-9826-5d378e75dd92 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:03.716148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cc6956-dbbf-4485-9826-5d378e75dd92 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:03.716445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cc6956-dbbf-4485-9826-5d378e75dd92 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:03.717812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cc6956-dbbf-4485-9826-5d378e75dd92 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:03.728835 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 339/637] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:03.729611 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cc6956-dbbf-4485-9826-5d378e75dd92 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:03.772810 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f34e08-188d-437b-a850-ee5b9caedda8 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:03.774352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:03.807886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9736b70d-a7e9-4d2b-8138-fbf6dbc91526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:03.808488 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9736b70d-a7e9-4d2b-8138-fbf6dbc91526 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:03.808759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9736b70d-a7e9-4d2b-8138-fbf6dbc91526 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:03.809026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9736b70d-a7e9-4d2b-8138-fbf6dbc91526 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:03.809709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9736b70d-a7e9-4d2b-8138-fbf6dbc91526 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:03.810005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9736b70d-a7e9-4d2b-8138-fbf6dbc91526 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:03.811475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9736b70d-a7e9-4d2b-8138-fbf6dbc91526 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:03.823265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 299/638] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:03.824407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5078288-3162-456e-a61e-64bac645e8e3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:03.824690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5078288-3162-456e-a61e-64bac645e8e3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:03.824965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5078288-3162-456e-a61e-64bac645e8e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:03.828591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cc6956-dbbf-4485-9826-5d378e75dd92 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:03.842042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-c5078288-3162-456e-a61e-64bac645e8e3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:03.843256 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c5078288-3162-456e-a61e-64bac645e8e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:03.863138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bedbf3-8d24-41fe-8b63-1e4416f3029f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:03.863741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bedbf3-8d24-41fe-8b63-1e4416f3029f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:03.864021 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bedbf3-8d24-41fe-8b63-1e4416f3029f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:03.864290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bedbf3-8d24-41fe-8b63-1e4416f3029f None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:03.864952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bedbf3-8d24-41fe-8b63-1e4416f3029f None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:03.865254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bedbf3-8d24-41fe-8b63-1e4416f3029f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:03.866641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bedbf3-8d24-41fe-8b63-1e4416f3029f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:03.874363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 340/639] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:27:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:03.875141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bedbf3-8d24-41fe-8b63-1e4416f3029f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:03.890377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:03.911251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 300/640] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:27:03.912098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5078288-3162-456e-a61e-64bac645e8e3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:03.981759 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bedbf3-8d24-41fe-8b63-1e4416f3029f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:04.011187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:04.016999 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d769b885-0401-4b19-955f-394f6017fff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:04.017630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d769b885-0401-4b19-955f-394f6017fff1 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:04.017908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d769b885-0401-4b19-955f-394f6017fff1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:04.018199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d769b885-0401-4b19-955f-394f6017fff1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:04.021040 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5078288-3162-456e-a61e-64bac645e8e3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:04.029141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d769b885-0401-4b19-955f-394f6017fff1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:04.033819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d769b885-0401-4b19-955f-394f6017fff1 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-d769b885-0401-4b19-955f-394f6017fff1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:04.044937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:04.046446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:04.047359 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:04.047359 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:04.047359 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:04.047359 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 341/641] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:04.047359 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d769b885-0401-4b19-955f-394f6017fff1 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:04.063470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f721e34-cd05-4ffc-bc7d-859fb718b844 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:04.064102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f721e34-cd05-4ffc-bc7d-859fb718b844 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:04.064381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f721e34-cd05-4ffc-bc7d-859fb718b844 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:04.064649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f721e34-cd05-4ffc-bc7d-859fb718b844 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:04.073459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f721e34-cd05-4ffc-bc7d-859fb718b844 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:04.074966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f721e34-cd05-4ffc-bc7d-859fb718b844 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-1f721e34-cd05-4ffc-bc7d-859fb718b844 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:04.088548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:04.089700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:04.090820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:04.090820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:04.090820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:04.090820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:04.090820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:04.090820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:04.090820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:04.090820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:04.095981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 301/642] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:04.097784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f721e34-cd05-4ffc-bc7d-859fb718b844 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:04.119462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:04.158469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d769b885-0401-4b19-955f-394f6017fff1 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:04.194271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69ed120-3dd7-4655-800a-abfc6cf1c38c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:04.194875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69ed120-3dd7-4655-800a-abfc6cf1c38c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:04.195162 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69ed120-3dd7-4655-800a-abfc6cf1c38c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:04.195574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69ed120-3dd7-4655-800a-abfc6cf1c38c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:04.196356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69ed120-3dd7-4655-800a-abfc6cf1c38c None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=e5896018712d4b82a781365e478ee855) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:04.196669 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69ed120-3dd7-4655-800a-abfc6cf1c38c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:04.198094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69ed120-3dd7-4655-800a-abfc6cf1c38c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:04.203972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 342/643] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:04 2022] GET /identity/v3/role_assignments?user.id=e5896018712d4b82a781365e478ee855&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:04.204754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69ed120-3dd7-4655-800a-abfc6cf1c38c None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:04.206536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f721e34-cd05-4ffc-bc7d-859fb718b844 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:04.247064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23b1b4c-f99c-4d27-ade3-5a3e4d738e55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:04.247618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23b1b4c-f99c-4d27-ade3-5a3e4d738e55 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:04.247897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23b1b4c-f99c-4d27-ade3-5a3e4d738e55 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:04.248160 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23b1b4c-f99c-4d27-ade3-5a3e4d738e55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:04.260310 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23b1b4c-f99c-4d27-ade3-5a3e4d738e55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:04.261797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23b1b4c-f99c-4d27-ade3-5a3e4d738e55 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-e23b1b4c-f99c-4d27-ade3-5a3e4d738e55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:04.277653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:04.278999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:04.280462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:04.280462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:04.280462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:04.280462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 302/644] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:04.280462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23b1b4c-f99c-4d27-ade3-5a3e4d738e55 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:04.289298 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:04.317710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69ed120-3dd7-4655-800a-abfc6cf1c38c None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:04.355187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359ae52e-141f-45f1-a7f3-54ec52156dd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:04.355787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359ae52e-141f-45f1-a7f3-54ec52156dd9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:04.356072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359ae52e-141f-45f1-a7f3-54ec52156dd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:04.356350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359ae52e-141f-45f1-a7f3-54ec52156dd9 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:04.357673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359ae52e-141f-45f1-a7f3-54ec52156dd9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:04.357673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359ae52e-141f-45f1-a7f3-54ec52156dd9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:04.359483 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359ae52e-141f-45f1-a7f3-54ec52156dd9 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:04.371288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 343/645] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:04.372924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359ae52e-141f-45f1-a7f3-54ec52156dd9 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:04.381196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:04.396725 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23b1b4c-f99c-4d27-ade3-5a3e4d738e55 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:04.438926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b127ec-f26e-4c60-b89f-8968fe3ca59a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:04.439547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b127ec-f26e-4c60-b89f-8968fe3ca59a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:04.439830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b127ec-f26e-4c60-b89f-8968fe3ca59a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:04.440101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b127ec-f26e-4c60-b89f-8968fe3ca59a None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:04.441079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b127ec-f26e-4c60-b89f-8968fe3ca59a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:04.441990 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b127ec-f26e-4c60-b89f-8968fe3ca59a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:04.444365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b127ec-f26e-4c60-b89f-8968fe3ca59a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:04.457647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 303/646] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:04.458592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b127ec-f26e-4c60-b89f-8968fe3ca59a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:04.495021 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359ae52e-141f-45f1-a7f3-54ec52156dd9 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:04.528641 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:04.530684 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8054152-1230-4ceb-be89-b52c272ab3b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:04.531315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8054152-1230-4ceb-be89-b52c272ab3b3 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:04.531591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8054152-1230-4ceb-be89-b52c272ab3b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:04.531863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8054152-1230-4ceb-be89-b52c272ab3b3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:04.542616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8054152-1230-4ceb-be89-b52c272ab3b3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-b8054152-1230-4ceb-be89-b52c272ab3b3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:04.547455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:04.548415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 344/647] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:27:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:04.549321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8054152-1230-4ceb-be89-b52c272ab3b3 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:04.574818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b127ec-f26e-4c60-b89f-8968fe3ca59a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:04.602711 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:04.612950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c22369-8360-46cb-9dc2-eef2562aefc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:04.613558 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c22369-8360-46cb-9dc2-eef2562aefc2 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:04.621434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c22369-8360-46cb-9dc2-eef2562aefc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:04.621434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c22369-8360-46cb-9dc2-eef2562aefc2 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:04.621434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c22369-8360-46cb-9dc2-eef2562aefc2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:04.621434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c22369-8360-46cb-9dc2-eef2562aefc2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:04.621434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c22369-8360-46cb-9dc2-eef2562aefc2 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:04.633085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 304/648] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:04.633997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c22369-8360-46cb-9dc2-eef2562aefc2 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:04.658433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8054152-1230-4ceb-be89-b52c272ab3b3 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:04.694590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbc4333-ac04-44c6-a40c-71ae59f0d390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:04.695242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbc4333-ac04-44c6-a40c-71ae59f0d390 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:04.695479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbc4333-ac04-44c6-a40c-71ae59f0d390 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:04.695748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbc4333-ac04-44c6-a40c-71ae59f0d390 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:04.702783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbc4333-ac04-44c6-a40c-71ae59f0d390 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:04.704221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbc4333-ac04-44c6-a40c-71ae59f0d390 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-6fbc4333-ac04-44c6-a40c-71ae59f0d390 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:04.715798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:04.716815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:04.717774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:04.717774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:04.717774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:04.717774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:04.717774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:04.717774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:04.717774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:04.717774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:04.717774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 24 03:27:04.717774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:04.717774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 345/649] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:27:04 2022] GET /identity/v3/users/glance => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:04.717774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbc4333-ac04-44c6-a40c-71ae59f0d390 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:04.748996 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c22369-8360-46cb-9dc2-eef2562aefc2 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:04.775202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:04.787698 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c753a3-7ed1-400f-ba7c-b08ea77735d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:04.788299 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c753a3-7ed1-400f-ba7c-b08ea77735d7 None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:04.788572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c753a3-7ed1-400f-ba7c-b08ea77735d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:04.788837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c753a3-7ed1-400f-ba7c-b08ea77735d7 None admin] PATH_INFO: `/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/4591bec74a4641af818b2d75b657115e/roles/94618a16f2e6449ba1cd5fbbd7c19f56` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:04.805227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c753a3-7ed1-400f-ba7c-b08ea77735d7 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c653d5fa4304f298a295ce635efd5e6, user_id=4591bec74a4641af818b2d75b657115e, role_id=94618a16f2e6449ba1cd5fbbd7c19f56)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:04.806712 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c753a3-7ed1-400f-ba7c-b08ea77735d7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:04.826591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbc4333-ac04-44c6-a40c-71ae59f0d390 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:04.836049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:04.859901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d5194d-9035-43bf-8d09-0224bf26f4a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:04.860503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d5194d-9035-43bf-8d09-0224bf26f4a9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:04.860781 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d5194d-9035-43bf-8d09-0224bf26f4a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:04.861049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d5194d-9035-43bf-8d09-0224bf26f4a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:04.861705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d5194d-9035-43bf-8d09-0224bf26f4a9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:04.862004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d5194d-9035-43bf-8d09-0224bf26f4a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:04.863879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d5194d-9035-43bf-8d09-0224bf26f4a9 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:04.872343 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 346/650] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:27:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:04.873235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d5194d-9035-43bf-8d09-0224bf26f4a9 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:04.909947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 305/651] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:27:04 2022] PUT /identity/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/4591bec74a4641af818b2d75b657115e/roles/94618a16f2e6449ba1cd5fbbd7c19f56 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:27:04.911166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c753a3-7ed1-400f-ba7c-b08ea77735d7 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:05.009283 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d5194d-9035-43bf-8d09-0224bf26f4a9 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:05.040424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c753a3-7ed1-400f-ba7c-b08ea77735d7 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:05.060034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335e40b9-ddac-4211-94f2-6e5643718627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:05.060937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335e40b9-ddac-4211-94f2-6e5643718627 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:05.061251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335e40b9-ddac-4211-94f2-6e5643718627 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:05.061529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335e40b9-ddac-4211-94f2-6e5643718627 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:05.072337 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335e40b9-ddac-4211-94f2-6e5643718627 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:05.074803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335e40b9-ddac-4211-94f2-6e5643718627 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:05.085740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b61297-63da-490f-b105-8bfaecb10bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:05.086446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b61297-63da-490f-b105-8bfaecb10bc9 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:05.086648 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b61297-63da-490f-b105-8bfaecb10bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:05.087285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b61297-63da-490f-b105-8bfaecb10bc9 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:05.087902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b61297-63da-490f-b105-8bfaecb10bc9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:05.087902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b61297-63da-490f-b105-8bfaecb10bc9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-335e40b9-ddac-4211-94f2-6e5643718627 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:05.088788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:05.089685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:05.090692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:05.090692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:05.090692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:05.090692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:05.090692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:05.090692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:05.090692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:05.090692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:05.090692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 24 03:27:05.090692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:05.091308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b61297-63da-490f-b105-8bfaecb10bc9 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:05.091379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 347/652] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:27:04 2022] GET /identity/v3/users/neutron => generated 85 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:05.091379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335e40b9-ddac-4211-94f2-6e5643718627 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:05.115395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 306/653] 10.222.0.65 () {56 vars in 1137 bytes} [Wed Aug 24 03:27:04 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:05.115395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b61297-63da-490f-b105-8bfaecb10bc9 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:05.130881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:05.153924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:05.207633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335e40b9-ddac-4211-94f2-6e5643718627 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:05.242923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b61297-63da-490f-b105-8bfaecb10bc9 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:05.246964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c198e9-060a-492f-b2f5-9dacd59389e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:05.246964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c198e9-060a-492f-b2f5-9dacd59389e6 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:05.246964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c198e9-060a-492f-b2f5-9dacd59389e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:05.246964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c198e9-060a-492f-b2f5-9dacd59389e6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:05.259148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c198e9-060a-492f-b2f5-9dacd59389e6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:05.261009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c198e9-060a-492f-b2f5-9dacd59389e6 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-87c198e9-060a-492f-b2f5-9dacd59389e6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:05.277600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:05.279432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:05.280326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:05.280326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:05.280326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:05.282354 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 348/654] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:05.283694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c198e9-060a-492f-b2f5-9dacd59389e6 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:05.284282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c96ffd-3fea-4030-9c62-fad1400e4716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:05.285006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c96ffd-3fea-4030-9c62-fad1400e4716 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:05.285329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c96ffd-3fea-4030-9c62-fad1400e4716 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:05.285551 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c96ffd-3fea-4030-9c62-fad1400e4716 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:05.286326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c96ffd-3fea-4030-9c62-fad1400e4716 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:05.286777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c96ffd-3fea-4030-9c62-fad1400e4716 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:05.288054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c96ffd-3fea-4030-9c62-fad1400e4716 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:05.311952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 307/655] 10.222.0.65 () {56 vars in 1139 bytes} [Wed Aug 24 03:27:05 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:05.313284 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c96ffd-3fea-4030-9c62-fad1400e4716 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:05.395362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:05.421361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c198e9-060a-492f-b2f5-9dacd59389e6 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:05.429455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:05.442926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c96ffd-3fea-4030-9c62-fad1400e4716 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:05.469298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dfb27a-225a-49d0-b591-6fdc75c298b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:05.470222 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dfb27a-225a-49d0-b591-6fdc75c298b1 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:05.470560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dfb27a-225a-49d0-b591-6fdc75c298b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:05.471090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dfb27a-225a-49d0-b591-6fdc75c298b1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:05.479829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ab7ea5-3df2-4524-9c95-cd46b973e522 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:05.480577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ab7ea5-3df2-4524-9c95-cd46b973e522 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:05.480923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ab7ea5-3df2-4524-9c95-cd46b973e522 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:05.481265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ab7ea5-3df2-4524-9c95-cd46b973e522 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:05.482009 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ab7ea5-3df2-4524-9c95-cd46b973e522 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:05.482409 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ab7ea5-3df2-4524-9c95-cd46b973e522 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:05.483840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ab7ea5-3df2-4524-9c95-cd46b973e522 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:05.501142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 308/656] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:05.502123 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ab7ea5-3df2-4524-9c95-cd46b973e522 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:05.561723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dfb27a-225a-49d0-b591-6fdc75c298b1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:05.563561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dfb27a-225a-49d0-b591-6fdc75c298b1 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:05.615853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ab7ea5-3df2-4524-9c95-cd46b973e522 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:05.647657 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:05.654408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b8bb63-657b-47ad-821c-c16b6879565c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:05.655194 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b8bb63-657b-47ad-821c-c16b6879565c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:05.655616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b8bb63-657b-47ad-821c-c16b6879565c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:05.667320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b8bb63-657b-47ad-821c-c16b6879565c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:05.668340 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 349/657] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:27:05.669191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dfb27a-225a-49d0-b591-6fdc75c298b1 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:05.677247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:05.730449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b8bb63-657b-47ad-821c-c16b6879565c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:05.732240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b8bb63-657b-47ad-821c-c16b6879565c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:05.778187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dfb27a-225a-49d0-b591-6fdc75c298b1 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:05.815962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa4e8b3-cd2c-4f30-a74b-69586f7a2d3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:05.816509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa4e8b3-cd2c-4f30-a74b-69586f7a2d3e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:05.816783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa4e8b3-cd2c-4f30-a74b-69586f7a2d3e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:05.817051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa4e8b3-cd2c-4f30-a74b-69586f7a2d3e None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:05.824764 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa4e8b3-cd2c-4f30-a74b-69586f7a2d3e None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:05.826509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa4e8b3-cd2c-4f30-a74b-69586f7a2d3e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:05.837592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 309/658] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:27:05.839087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b8bb63-657b-47ad-821c-c16b6879565c None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-cfa4e8b3-cd2c-4f30-a74b-69586f7a2d3e None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:05.839419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:05.840342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:05.841289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:05.841289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:05.841289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:05.841289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:05.841289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:05.841289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:05.841289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:05.841289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:05.841289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 03:27:05.841289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:05.846045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 350/659] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:27:05 2022] GET /identity/v3/users/cinder => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:05.846216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa4e8b3-cd2c-4f30-a74b-69586f7a2d3e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:05.894049 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:05.899534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:05.946384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b8bb63-657b-47ad-821c-c16b6879565c None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:05.960963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa4e8b3-cd2c-4f30-a74b-69586f7a2d3e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:05.983757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a67235e-e022-41f7-820d-12304ab2e7bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:05.984767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a67235e-e022-41f7-820d-12304ab2e7bd None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:05.985255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a67235e-e022-41f7-820d-12304ab2e7bd None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:05.985787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a67235e-e022-41f7-820d-12304ab2e7bd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:05.997438 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a67235e-e022-41f7-820d-12304ab2e7bd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:05.999249 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a67235e-e022-41f7-820d-12304ab2e7bd None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:06.007410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07be1d42-f1b4-4c49-9ba3-09b73caeae5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:06.008042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07be1d42-f1b4-4c49-9ba3-09b73caeae5e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:06.008318 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07be1d42-f1b4-4c49-9ba3-09b73caeae5e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:06.008586 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07be1d42-f1b4-4c49-9ba3-09b73caeae5e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-0a67235e-e022-41f7-820d-12304ab2e7bd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:06.016858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:06.017807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:06.019287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:06.019287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:06.019287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:06.019287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 310/660] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:06.019976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a67235e-e022-41f7-820d-12304ab2e7bd None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:06.032911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07be1d42-f1b4-4c49-9ba3-09b73caeae5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:06.035176 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07be1d42-f1b4-4c49-9ba3-09b73caeae5e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-07be1d42-f1b4-4c49-9ba3-09b73caeae5e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:06.050445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:06.051885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:06.053057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:06.053057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:06.053057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:06.053057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 351/661] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:06.053057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07be1d42-f1b4-4c49-9ba3-09b73caeae5e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:06.158507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:06.160499 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a67235e-e022-41f7-820d-12304ab2e7bd None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:06.162375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:06.167103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07be1d42-f1b4-4c49-9ba3-09b73caeae5e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:06.201512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0fff06-f42c-4ed4-a36c-cec5c1f09ed2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:06.202416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0fff06-f42c-4ed4-a36c-cec5c1f09ed2 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:06.202670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0fff06-f42c-4ed4-a36c-cec5c1f09ed2 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:06.202942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0fff06-f42c-4ed4-a36c-cec5c1f09ed2 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:06.203665 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0fff06-f42c-4ed4-a36c-cec5c1f09ed2 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:06.203963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0fff06-f42c-4ed4-a36c-cec5c1f09ed2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:06.205455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0fff06-f42c-4ed4-a36c-cec5c1f09ed2 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:06.206517 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fdde84-2b38-4e96-96f3-b2aedc808063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:06.206961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fdde84-2b38-4e96-96f3-b2aedc808063 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:06.207319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fdde84-2b38-4e96-96f3-b2aedc808063 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:06.207757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fdde84-2b38-4e96-96f3-b2aedc808063 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:06.209154 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fdde84-2b38-4e96-96f3-b2aedc808063 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:06.209478 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fdde84-2b38-4e96-96f3-b2aedc808063 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:06.210859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fdde84-2b38-4e96-96f3-b2aedc808063 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:06.226883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 352/662] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:06.227410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 311/663] 10.222.0.65 () {56 vars in 1137 bytes} [Wed Aug 24 03:27:06 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:06.227768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fdde84-2b38-4e96-96f3-b2aedc808063 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:06.232679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0fff06-f42c-4ed4-a36c-cec5c1f09ed2 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:06.337513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0fff06-f42c-4ed4-a36c-cec5c1f09ed2 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:06.342202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fdde84-2b38-4e96-96f3-b2aedc808063 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:06.375680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f3eeb6-fbc2-4214-a59e-b7c20d9dd7da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:06.376396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3eeb6-fbc2-4214-a59e-b7c20d9dd7da None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:06.376734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3eeb6-fbc2-4214-a59e-b7c20d9dd7da None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:06.377083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3eeb6-fbc2-4214-a59e-b7c20d9dd7da None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:06.377329 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a58c12-71eb-4be5-a705-b2d349491a8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:06.378534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a58c12-71eb-4be5-a705-b2d349491a8f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:06.378886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a58c12-71eb-4be5-a705-b2d349491a8f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:06.379224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a58c12-71eb-4be5-a705-b2d349491a8f None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:06.379936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a58c12-71eb-4be5-a705-b2d349491a8f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:06.380304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a58c12-71eb-4be5-a705-b2d349491a8f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:06.381987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a58c12-71eb-4be5-a705-b2d349491a8f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:06.385622 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f3eeb6-fbc2-4214-a59e-b7c20d9dd7da None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:06.387147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f3eeb6-fbc2-4214-a59e-b7c20d9dd7da None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:06.398357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 353/664] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:06.398357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a58c12-71eb-4be5-a705-b2d349491a8f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-67f3eeb6-fbc2-4214-a59e-b7c20d9dd7da None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:06.398870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:06.399882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:06.401005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:06.401005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:06.401005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:06.401005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:06.401005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:06.401005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:06.401005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:06.401005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:06.406415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 312/665] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:06.406415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d32a0d-1979-4684-966e-82771aa07b55 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:06.406415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d32a0d-1979-4684-966e-82771aa07b55 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:06.406415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d32a0d-1979-4684-966e-82771aa07b55 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:06.406415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 313/666] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:06 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:06.406415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d32a0d-1979-4684-966e-82771aa07b55 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:06.423831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:06.438031 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:06.512968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a58c12-71eb-4be5-a705-b2d349491a8f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:06.520367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d32a0d-1979-4684-966e-82771aa07b55 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:06.549382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2e5ebe-b577-4b26-9e04-f3f19bb63c9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:06.551058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2e5ebe-b577-4b26-9e04-f3f19bb63c9c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:06.551348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2e5ebe-b577-4b26-9e04-f3f19bb63c9c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:06.551619 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2e5ebe-b577-4b26-9e04-f3f19bb63c9c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:06.556387 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574643ff-04f3-4208-922d-6635014d64d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:06.557307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574643ff-04f3-4208-922d-6635014d64d5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:06.557786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574643ff-04f3-4208-922d-6635014d64d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:06.558303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574643ff-04f3-4208-922d-6635014d64d5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:06.565922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574643ff-04f3-4208-922d-6635014d64d5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:06.567705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574643ff-04f3-4208-922d-6635014d64d5 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-574643ff-04f3-4208-922d-6635014d64d5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:06.579382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:06.580278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:06.581392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:06.581392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:06.581392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:06.581392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:06.581392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:06.581392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:06.581392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:06.581392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:06.581392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 314/667] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:06.581879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574643ff-04f3-4208-922d-6635014d64d5 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:06.625163 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2e5ebe-b577-4b26-9e04-f3f19bb63c9c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:06.626673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2e5ebe-b577-4b26-9e04-f3f19bb63c9c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:06.642803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:06.673640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:06.695094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574643ff-04f3-4208-922d-6635014d64d5 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:06.728840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 354/668] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:27:06.730519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28acfdd-5c01-4151-9394-420fe7c6f921 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:06.731020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28acfdd-5c01-4151-9394-420fe7c6f921 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:06.731500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28acfdd-5c01-4151-9394-420fe7c6f921 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:06.734530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144f331f-8079-46a9-9ebc-d4d0c73e7878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:06.735191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144f331f-8079-46a9-9ebc-d4d0c73e7878 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:06.735473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144f331f-8079-46a9-9ebc-d4d0c73e7878 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:06.735740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144f331f-8079-46a9-9ebc-d4d0c73e7878 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:06.736423 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144f331f-8079-46a9-9ebc-d4d0c73e7878 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:06.736719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144f331f-8079-46a9-9ebc-d4d0c73e7878 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:06.738934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144f331f-8079-46a9-9ebc-d4d0c73e7878 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:06.758342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 315/669] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:06.758342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144f331f-8079-46a9-9ebc-d4d0c73e7878 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:06.759643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c28acfdd-5c01-4151-9394-420fe7c6f921 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:06.760859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c28acfdd-5c01-4151-9394-420fe7c6f921 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:06.846191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 355/670] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:27:06.846191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28acfdd-5c01-4151-9394-420fe7c6f921 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:06.877125 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144f331f-8079-46a9-9ebc-d4d0c73e7878 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:06.906919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:06.917912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7434cd5-f289-4ff4-88f3-e425c5009e44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:06.923575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7434cd5-f289-4ff4-88f3-e425c5009e44 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:06.923575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7434cd5-f289-4ff4-88f3-e425c5009e44 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:06.923575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7434cd5-f289-4ff4-88f3-e425c5009e44 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:06.923575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7434cd5-f289-4ff4-88f3-e425c5009e44 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:06.923575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7434cd5-f289-4ff4-88f3-e425c5009e44 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:06.923575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7434cd5-f289-4ff4-88f3-e425c5009e44 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:06.931045 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:06.934824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 316/671] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:06.935749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7434cd5-f289-4ff4-88f3-e425c5009e44 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:06.974991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28acfdd-5c01-4151-9394-420fe7c6f921 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:07.011994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad72c410-2da0-4ea1-bd86-686c53de488b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:07.012829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad72c410-2da0-4ea1-bd86-686c53de488b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:07.013113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad72c410-2da0-4ea1-bd86-686c53de488b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:07.013383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad72c410-2da0-4ea1-bd86-686c53de488b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:07.024412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad72c410-2da0-4ea1-bd86-686c53de488b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:07.025863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad72c410-2da0-4ea1-bd86-686c53de488b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:07.038987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7434cd5-f289-4ff4-88f3-e425c5009e44 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-ad72c410-2da0-4ea1-bd86-686c53de488b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:07.041717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:07.042662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:07.043616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:07.043616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:07.043616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:07.047354 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 356/672] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:07.048052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad72c410-2da0-4ea1-bd86-686c53de488b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:07.073946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a1c409-8493-457a-90c8-6ae13c3babbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:07.074585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a1c409-8493-457a-90c8-6ae13c3babbf None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:07.074907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a1c409-8493-457a-90c8-6ae13c3babbf None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:07.075130 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a1c409-8493-457a-90c8-6ae13c3babbf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:07.075848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a1c409-8493-457a-90c8-6ae13c3babbf None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=9f3b3c7ebc08496e970f8f25fd6395e4) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:07.076154 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a1c409-8493-457a-90c8-6ae13c3babbf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:07.077597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a1c409-8493-457a-90c8-6ae13c3babbf None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:07.090192 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 317/673] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:06 2022] GET /identity/v3/role_assignments?user.id=9f3b3c7ebc08496e970f8f25fd6395e4&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:07.090192 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83d470d-5a88-41ed-9c26-1854b719c113 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:07.090192 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83d470d-5a88-41ed-9c26-1854b719c113 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:07.090192 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83d470d-5a88-41ed-9c26-1854b719c113 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:07.093037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 318/674] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:07 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:07.093037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83d470d-5a88-41ed-9c26-1854b719c113 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:07.160686 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:07.166815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:07.168776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad72c410-2da0-4ea1-bd86-686c53de488b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:07.209045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750d445e-4a2f-41fb-aa20-e838089312c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:07.209576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750d445e-4a2f-41fb-aa20-e838089312c5 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:07.209851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750d445e-4a2f-41fb-aa20-e838089312c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:07.210121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750d445e-4a2f-41fb-aa20-e838089312c5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:07.210528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83d470d-5a88-41ed-9c26-1854b719c113 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:07.210895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750d445e-4a2f-41fb-aa20-e838089312c5 None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=64f23f0c8abc406b9700361723623387) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:07.211204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750d445e-4a2f-41fb-aa20-e838089312c5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:07.212764 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750d445e-4a2f-41fb-aa20-e838089312c5 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:07.219453 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 357/675] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:07 2022] GET /identity/v3/role_assignments?user.id=64f23f0c8abc406b9700361723623387&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:07.220446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05c87bf-73f2-442b-8cb6-27ad88bf1e70 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:07.220732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05c87bf-73f2-442b-8cb6-27ad88bf1e70 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:07.221005 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05c87bf-73f2-442b-8cb6-27ad88bf1e70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:07.242812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d05c87bf-73f2-442b-8cb6-27ad88bf1e70 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:07.244202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d05c87bf-73f2-442b-8cb6-27ad88bf1e70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:07.260496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b559d057-814d-47a3-b774-e1ca4e8ac180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:07.261108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b559d057-814d-47a3-b774-e1ca4e8ac180 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:07.261382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b559d057-814d-47a3-b774-e1ca4e8ac180 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:07.261651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b559d057-814d-47a3-b774-e1ca4e8ac180 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:07.262327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b559d057-814d-47a3-b774-e1ca4e8ac180 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:07.262624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b559d057-814d-47a3-b774-e1ca4e8ac180 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:07.264105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b559d057-814d-47a3-b774-e1ca4e8ac180 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:07.281608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 319/676] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:07.282991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b559d057-814d-47a3-b774-e1ca4e8ac180 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:07.348383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 358/677] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:27:07.348581 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05c87bf-73f2-442b-8cb6-27ad88bf1e70 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:07.407806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b559d057-814d-47a3-b774-e1ca4e8ac180 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:07.444398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:07.456429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c094ba-958e-4335-b5b0-82194cf2bdee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:07.457973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c094ba-958e-4335-b5b0-82194cf2bdee None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:07.457973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c094ba-958e-4335-b5b0-82194cf2bdee None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:07.457973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c094ba-958e-4335-b5b0-82194cf2bdee None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:07.458747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:07.471306 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c094ba-958e-4335-b5b0-82194cf2bdee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:07.473249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05c87bf-73f2-442b-8cb6-27ad88bf1e70 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-10c094ba-958e-4335-b5b0-82194cf2bdee None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:07.477606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:07.478583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:07.479765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 320/678] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:07.485390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c094ba-958e-4335-b5b0-82194cf2bdee None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:07.520307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3630b1da-d090-4b3e-ae69-7203541dacfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:07.521529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3630b1da-d090-4b3e-ae69-7203541dacfd None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:07.522302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3630b1da-d090-4b3e-ae69-7203541dacfd None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:07.522763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3630b1da-d090-4b3e-ae69-7203541dacfd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:07.531598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3630b1da-d090-4b3e-ae69-7203541dacfd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:07.534243 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3630b1da-d090-4b3e-ae69-7203541dacfd None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-3630b1da-d090-4b3e-ae69-7203541dacfd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:07.547268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:07.548378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:07.550040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:07.550040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:07.550040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:07.550040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:07.550040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:07.550040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:07.550040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:07.550040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:07.551076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 359/679] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:07.561621 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3630b1da-d090-4b3e-ae69-7203541dacfd None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:07.657219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c094ba-958e-4335-b5b0-82194cf2bdee None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:07.685076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3630b1da-d090-4b3e-ae69-7203541dacfd None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:07.695733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c040b4-a396-4141-9a13-137829d9c674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:07.696344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c040b4-a396-4141-9a13-137829d9c674 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:07.696616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c040b4-a396-4141-9a13-137829d9c674 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:07.696913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c040b4-a396-4141-9a13-137829d9c674 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:07.697643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c040b4-a396-4141-9a13-137829d9c674 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:07.697943 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c040b4-a396-4141-9a13-137829d9c674 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:07.699341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c040b4-a396-4141-9a13-137829d9c674 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:07.709337 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 321/680] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:27:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:07.710189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c040b4-a396-4141-9a13-137829d9c674 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:07.728584 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ccd8ce-7651-4518-811f-c453c539be80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:07.729122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ccd8ce-7651-4518-811f-c453c539be80 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:07.729401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ccd8ce-7651-4518-811f-c453c539be80 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:07.729734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ccd8ce-7651-4518-811f-c453c539be80 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:07.730578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ccd8ce-7651-4518-811f-c453c539be80 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:07.731027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ccd8ce-7651-4518-811f-c453c539be80 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:07.732521 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ccd8ce-7651-4518-811f-c453c539be80 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:07.746363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 360/681] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:07.747399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ccd8ce-7651-4518-811f-c453c539be80 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:07.760744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:07.762794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:07.817987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c040b4-a396-4141-9a13-137829d9c674 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:07.855831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4f3d65-f68d-40d2-b572-26594228a6e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:07.856875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4f3d65-f68d-40d2-b572-26594228a6e6 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:07.857361 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4f3d65-f68d-40d2-b572-26594228a6e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:07.857840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4f3d65-f68d-40d2-b572-26594228a6e6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:07.863221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ccd8ce-7651-4518-811f-c453c539be80 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:07.871316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4f3d65-f68d-40d2-b572-26594228a6e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-8e4f3d65-f68d-40d2-b572-26594228a6e6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:07.876977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:07.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:07.879779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:07.879779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:07.879779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:07.879779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:07.879779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:07.879779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:07.879779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:07.879779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:07.879779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:07.879779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:07.879779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:07.879779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:07.886875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 322/682] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:27:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:07.887900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4f3d65-f68d-40d2-b572-26594228a6e6 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:07.908104 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9025aa-d26e-434e-b749-7298d2e7b240 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:07.908739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9025aa-d26e-434e-b749-7298d2e7b240 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:07.909578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9025aa-d26e-434e-b749-7298d2e7b240 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:07.909853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9025aa-d26e-434e-b749-7298d2e7b240 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:07.925950 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9025aa-d26e-434e-b749-7298d2e7b240 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:07.927504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9025aa-d26e-434e-b749-7298d2e7b240 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-fe9025aa-d26e-434e-b749-7298d2e7b240 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:07.946968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:07.947979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:07.948882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:07.948882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:07.948882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:07.948882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 361/683] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:07.949589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9025aa-d26e-434e-b749-7298d2e7b240 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:08.001145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4f3d65-f68d-40d2-b572-26594228a6e6 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:08.010397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:08.038311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c17ceca-1d16-46f7-9989-6fd6f1e0fb58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:08.039003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:08.040528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c17ceca-1d16-46f7-9989-6fd6f1e0fb58 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:08.041008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c17ceca-1d16-46f7-9989-6fd6f1e0fb58 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:08.041538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c17ceca-1d16-46f7-9989-6fd6f1e0fb58 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:08.042505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c17ceca-1d16-46f7-9989-6fd6f1e0fb58 None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=dbfbfd64573141f88040692119ac5a1f) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:08.043013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c17ceca-1d16-46f7-9989-6fd6f1e0fb58 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:08.044692 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c17ceca-1d16-46f7-9989-6fd6f1e0fb58 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:08.053230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 323/684] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:07 2022] GET /identity/v3/role_assignments?user.id=dbfbfd64573141f88040692119ac5a1f&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:08.054415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c17ceca-1d16-46f7-9989-6fd6f1e0fb58 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:08.116184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9025aa-d26e-434e-b749-7298d2e7b240 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:08.210559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c17ceca-1d16-46f7-9989-6fd6f1e0fb58 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:08.216923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8081d0e3-d572-4b28-b793-7338c89b2068 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:08.217939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8081d0e3-d572-4b28-b793-7338c89b2068 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:08.218453 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8081d0e3-d572-4b28-b793-7338c89b2068 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:08.218938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8081d0e3-d572-4b28-b793-7338c89b2068 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:08.219821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8081d0e3-d572-4b28-b793-7338c89b2068 None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:08.220327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8081d0e3-d572-4b28-b793-7338c89b2068 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:08.221933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8081d0e3-d572-4b28-b793-7338c89b2068 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:08.244619 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 362/685] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:27:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:08.249700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df5d68a-af52-44ad-a913-62c49857706f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:08.250850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8081d0e3-d572-4b28-b793-7338c89b2068 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:08.254480 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df5d68a-af52-44ad-a913-62c49857706f None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:08.255119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df5d68a-af52-44ad-a913-62c49857706f None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:08.255866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df5d68a-af52-44ad-a913-62c49857706f None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:08.257603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df5d68a-af52-44ad-a913-62c49857706f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:08.259662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df5d68a-af52-44ad-a913-62c49857706f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:08.262619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df5d68a-af52-44ad-a913-62c49857706f None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:08.283147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 324/686] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:08.289052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df5d68a-af52-44ad-a913-62c49857706f None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:08.334266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:08.405461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8081d0e3-d572-4b28-b793-7338c89b2068 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:08.431250 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df5d68a-af52-44ad-a913-62c49857706f None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:08.435692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:08.445452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f42a1b-b192-4c73-b752-b5a840f216b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:08.448077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f42a1b-b192-4c73-b752-b5a840f216b0 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:08.448077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f42a1b-b192-4c73-b752-b5a840f216b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:08.448077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f42a1b-b192-4c73-b752-b5a840f216b0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:08.460119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f42a1b-b192-4c73-b752-b5a840f216b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:08.461750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f42a1b-b192-4c73-b752-b5a840f216b0 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:08.474853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f40835d-004d-4a96-b5e3-10ecdff21c92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:08.476219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f40835d-004d-4a96-b5e3-10ecdff21c92 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:08.476504 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f40835d-004d-4a96-b5e3-10ecdff21c92 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:08.476829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f40835d-004d-4a96-b5e3-10ecdff21c92 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-91f42a1b-b192-4c73-b752-b5a840f216b0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:08.480441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:08.481376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:08.482351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:08.482351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:08.482351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:08.482351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 363/687] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:08.485359 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f40835d-004d-4a96-b5e3-10ecdff21c92 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:08.488294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f40835d-004d-4a96-b5e3-10ecdff21c92 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:08.491148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f42a1b-b192-4c73-b752-b5a840f216b0 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-9f40835d-004d-4a96-b5e3-10ecdff21c92 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:08.500815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:08.501842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:08.502898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:08.502898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:08.502898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:08.502898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:08.502898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:08.502898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:08.502898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:08.502898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:08.502898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:08.502898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:08.505979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 325/688] 10.222.0.65 () {56 vars in 1093 bytes} [Wed Aug 24 03:27:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:08.511253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f40835d-004d-4a96-b5e3-10ecdff21c92 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:08.630518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:08.641050 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f40835d-004d-4a96-b5e3-10ecdff21c92 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:08.662542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f42a1b-b192-4c73-b752-b5a840f216b0 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:08.700505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bed4bd3-d75b-43c9-b3ef-08bfcb9003a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:08.701149 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bed4bd3-d75b-43c9-b3ef-08bfcb9003a1 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:08.701439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bed4bd3-d75b-43c9-b3ef-08bfcb9003a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:08.701716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bed4bd3-d75b-43c9-b3ef-08bfcb9003a1 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:08.702460 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bed4bd3-d75b-43c9-b3ef-08bfcb9003a1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:08.702767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bed4bd3-d75b-43c9-b3ef-08bfcb9003a1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:08.704196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bed4bd3-d75b-43c9-b3ef-08bfcb9003a1 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:08.717440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 364/689] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:08.724077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bed4bd3-d75b-43c9-b3ef-08bfcb9003a1 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:08.769403 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:08.770862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a535a99-5f42-4798-8ff0-12e88a37ec02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:08.771484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a535a99-5f42-4798-8ff0-12e88a37ec02 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:08.771762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a535a99-5f42-4798-8ff0-12e88a37ec02 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:08.772032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a535a99-5f42-4798-8ff0-12e88a37ec02 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:08.772756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a535a99-5f42-4798-8ff0-12e88a37ec02 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:08.773060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a535a99-5f42-4798-8ff0-12e88a37ec02 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:08.774511 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a535a99-5f42-4798-8ff0-12e88a37ec02 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:08.810215 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 326/690] 10.222.0.65 () {56 vars in 1133 bytes} [Wed Aug 24 03:27:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:08.817237 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a535a99-5f42-4798-8ff0-12e88a37ec02 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:08.860725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bed4bd3-d75b-43c9-b3ef-08bfcb9003a1 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:08.927285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a535a99-5f42-4798-8ff0-12e88a37ec02 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:08.929341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7df8bbc-a2e4-41a7-8575-2db67c234b5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:08.930085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7df8bbc-a2e4-41a7-8575-2db67c234b5c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:08.930397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7df8bbc-a2e4-41a7-8575-2db67c234b5c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:08.930675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7df8bbc-a2e4-41a7-8575-2db67c234b5c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:08.947312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7df8bbc-a2e4-41a7-8575-2db67c234b5c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:08.948902 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7df8bbc-a2e4-41a7-8575-2db67c234b5c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-c7df8bbc-a2e4-41a7-8575-2db67c234b5c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:08.968514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:08.969518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:08.970580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:08.970580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:08.970580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:08.970580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:08.970580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:08.970580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:08.970580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:08.970580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:08.970580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 24 03:27:08.970580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:08.970580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 365/691] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:08 2022] GET /identity/v3/users/swift => generated 83 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:08.974088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ea7397-c3ec-4f65-a314-d22b7f59e392 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:08.974743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ea7397-c3ec-4f65-a314-d22b7f59e392 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:08.975077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ea7397-c3ec-4f65-a314-d22b7f59e392 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:08.975907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ea7397-c3ec-4f65-a314-d22b7f59e392 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:08.979173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7df8bbc-a2e4-41a7-8575-2db67c234b5c None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:09.014705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:09.080565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ea7397-c3ec-4f65-a314-d22b7f59e392 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:09.083147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ea7397-c3ec-4f65-a314-d22b7f59e392 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:09.130918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:09.133699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7df8bbc-a2e4-41a7-8575-2db67c234b5c None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:09.169943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9e0a1c-d87a-427b-9df6-d8c5448cc963 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:09.170586 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9e0a1c-d87a-427b-9df6-d8c5448cc963 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:09.170916 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9e0a1c-d87a-427b-9df6-d8c5448cc963 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:09.171241 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9e0a1c-d87a-427b-9df6-d8c5448cc963 None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:09.174175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9e0a1c-d87a-427b-9df6-d8c5448cc963 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:09.174175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9e0a1c-d87a-427b-9df6-d8c5448cc963 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:09.174175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9e0a1c-d87a-427b-9df6-d8c5448cc963 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:09.193543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 366/692] 10.222.0.65 () {56 vars in 1135 bytes} [Wed Aug 24 03:27:08 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:09.199101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9e0a1c-d87a-427b-9df6-d8c5448cc963 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:09.283468 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 327/693] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:27:09.289904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ea7397-c3ec-4f65-a314-d22b7f59e392 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:09.318867 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9e0a1c-d87a-427b-9df6-d8c5448cc963 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:09.359313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cca979c-9e91-4c23-8c8f-94196b966716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:09.360055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cca979c-9e91-4c23-8c8f-94196b966716 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:09.360342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cca979c-9e91-4c23-8c8f-94196b966716 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:09.360616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cca979c-9e91-4c23-8c8f-94196b966716 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:09.390264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:09.401977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:09.448181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cca979c-9e91-4c23-8c8f-94196b966716 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:09.450184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cca979c-9e91-4c23-8c8f-94196b966716 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:09.470128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ea7397-c3ec-4f65-a314-d22b7f59e392 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:09.516263 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3623df-088e-46ee-bb31-dbaa2ad1d361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:09.516972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3623df-088e-46ee-bb31-dbaa2ad1d361 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:09.517251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3623df-088e-46ee-bb31-dbaa2ad1d361 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:09.517520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3623df-088e-46ee-bb31-dbaa2ad1d361 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:09.529542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3623df-088e-46ee-bb31-dbaa2ad1d361 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:09.531005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3623df-088e-46ee-bb31-dbaa2ad1d361 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-9d3623df-088e-46ee-bb31-dbaa2ad1d361 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:09.547175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:09.548170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:09.549072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:09.549072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:09.549072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:09.553359 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 328/694] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:09.554792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4823a502-8cd2-4ccb-9c9a-cb1ff64def9f None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:09.555169 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4823a502-8cd2-4ccb-9c9a-cb1ff64def9f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:09.555442 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4823a502-8cd2-4ccb-9c9a-cb1ff64def9f None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:09.557063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 329/695] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:09.557758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4823a502-8cd2-4ccb-9c9a-cb1ff64def9f None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:09.559624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 367/696] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 24 03:27:09.565671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cca979c-9e91-4c23-8c8f-94196b966716 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:09.624036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:09.670293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cca979c-9e91-4c23-8c8f-94196b966716 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:09.670620 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:09.675512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4823a502-8cd2-4ccb-9c9a-cb1ff64def9f None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:09.707082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2110323b-70f4-41ac-9c14-a50a20f32072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:09.707766 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2110323b-70f4-41ac-9c14-a50a20f32072 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:09.708045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2110323b-70f4-41ac-9c14-a50a20f32072 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:09.708312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2110323b-70f4-41ac-9c14-a50a20f32072 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:09.713858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dbddf6-223f-4229-9771-f3b91ba468f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:09.714484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dbddf6-223f-4229-9771-f3b91ba468f7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:09.714759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dbddf6-223f-4229-9771-f3b91ba468f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:09.715049 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dbddf6-223f-4229-9771-f3b91ba468f7 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:09.715747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dbddf6-223f-4229-9771-f3b91ba468f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:09.716047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dbddf6-223f-4229-9771-f3b91ba468f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:09.717499 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dbddf6-223f-4229-9771-f3b91ba468f7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:09.719137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2110323b-70f4-41ac-9c14-a50a20f32072 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:09.720630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2110323b-70f4-41ac-9c14-a50a20f32072 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:09.735311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 330/697] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:09.736594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476cb767-c1a0-4233-8668-20d57432f26b None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-2110323b-70f4-41ac-9c14-a50a20f32072 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:09.736770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:09.739393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:09.741509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:09.741509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:09.741509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:09.741906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476cb767-c1a0-4233-8668-20d57432f26b None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:09.741906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476cb767-c1a0-4233-8668-20d57432f26b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:09.746415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 368/698] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:09.746415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2110323b-70f4-41ac-9c14-a50a20f32072 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:09.755832 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-476cb767-c1a0-4233-8668-20d57432f26b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:09.757030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-476cb767-c1a0-4233-8668-20d57432f26b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:09.827192 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 331/699] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:27:09.828410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476cb767-c1a0-4233-8668-20d57432f26b None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:09.862251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2110323b-70f4-41ac-9c14-a50a20f32072 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:09.870475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:09.900187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a5dc82-959d-4e9d-8e17-77c28099553b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:09.901053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a5dc82-959d-4e9d-8e17-77c28099553b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:09.901479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a5dc82-959d-4e9d-8e17-77c28099553b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:09.902012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a5dc82-959d-4e9d-8e17-77c28099553b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:09.910812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a5dc82-959d-4e9d-8e17-77c28099553b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:09.912387 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a5dc82-959d-4e9d-8e17-77c28099553b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-87a5dc82-959d-4e9d-8e17-77c28099553b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:09.924242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:09.925438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:09.926385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:09.926385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:09.926385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:09.926385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:09.926385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:09.926385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:09.926385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:09.926385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:09.930512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 369/700] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:09.931779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f0df81-d9ef-4f78-95b9-d9e95a9bb063 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:09.932060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f0df81-d9ef-4f78-95b9-d9e95a9bb063 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:09.932332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f0df81-d9ef-4f78-95b9-d9e95a9bb063 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:09.937596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 370/701] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:09.937596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f0df81-d9ef-4f78-95b9-d9e95a9bb063 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:09.955035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:09.967703 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476cb767-c1a0-4233-8668-20d57432f26b None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:10.014454 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cf5d26-ea37-4c84-a035-0e30ae490bf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:10.015447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cf5d26-ea37-4c84-a035-0e30ae490bf5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:10.015941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cf5d26-ea37-4c84-a035-0e30ae490bf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:10.016411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cf5d26-ea37-4c84-a035-0e30ae490bf5 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:10.017691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cf5d26-ea37-4c84-a035-0e30ae490bf5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:10.018327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cf5d26-ea37-4c84-a035-0e30ae490bf5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:10.020376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cf5d26-ea37-4c84-a035-0e30ae490bf5 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:10.036479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 332/702] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:10.038614 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4de9943-4cbc-4bc6-bb71-43b606194a50 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:10.039391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4de9943-4cbc-4bc6-bb71-43b606194a50 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:10.039889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4de9943-4cbc-4bc6-bb71-43b606194a50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:10.044875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f0df81-d9ef-4f78-95b9-d9e95a9bb063 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:10.064113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-e4de9943-4cbc-4bc6-bb71-43b606194a50 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:10.066960 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-e4de9943-4cbc-4bc6-bb71-43b606194a50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:10.080120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c4c003-caf0-45ea-8884-e1a298cadfae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:10.080715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c4c003-caf0-45ea-8884-e1a298cadfae None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:10.081783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c4c003-caf0-45ea-8884-e1a298cadfae None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:10.082310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c4c003-caf0-45ea-8884-e1a298cadfae None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:10.083042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c4c003-caf0-45ea-8884-e1a298cadfae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:10.083344 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c4c003-caf0-45ea-8884-e1a298cadfae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:10.084763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c4c003-caf0-45ea-8884-e1a298cadfae None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:10.102431 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 371/703] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:10.103294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c4c003-caf0-45ea-8884-e1a298cadfae None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:10.133022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:10.166317 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 333/704] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:27:10.167190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4de9943-4cbc-4bc6-bb71-43b606194a50 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:10.237690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c4c003-caf0-45ea-8884-e1a298cadfae None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:10.260241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:10.280451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4de9943-4cbc-4bc6-bb71-43b606194a50 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:10.285630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29787f28-7b64-4cd3-86c4-91e822a6dccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:10.286292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29787f28-7b64-4cd3-86c4-91e822a6dccc None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:10.286573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29787f28-7b64-4cd3-86c4-91e822a6dccc None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:10.286844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29787f28-7b64-4cd3-86c4-91e822a6dccc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:10.296276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29787f28-7b64-4cd3-86c4-91e822a6dccc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:10.298117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29787f28-7b64-4cd3-86c4-91e822a6dccc None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-29787f28-7b64-4cd3-86c4-91e822a6dccc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:10.313791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:10.314954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:10.317224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:10.317224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:10.317224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:10.317224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:10.317224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:10.317224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:10.317224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:10.317224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:10.320413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 372/705] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:10.321807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29787f28-7b64-4cd3-86c4-91e822a6dccc None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:10.326857 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cadb53c-ece3-49ab-8242-b32d12827efb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:10.327731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cadb53c-ece3-49ab-8242-b32d12827efb None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:10.328186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cadb53c-ece3-49ab-8242-b32d12827efb None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:10.328632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cadb53c-ece3-49ab-8242-b32d12827efb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:10.329528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cadb53c-ece3-49ab-8242-b32d12827efb None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=e5896018712d4b82a781365e478ee855) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:10.330055 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cadb53c-ece3-49ab-8242-b32d12827efb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:10.332546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cadb53c-ece3-49ab-8242-b32d12827efb None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:10.343443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 334/706] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:10 2022] GET /identity/v3/role_assignments?user.id=e5896018712d4b82a781365e478ee855&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=43ca25467bc6484da3a50508efb79230 => generated 259 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:10.344858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cadb53c-ece3-49ab-8242-b32d12827efb None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:10.429899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:10.449438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29787f28-7b64-4cd3-86c4-91e822a6dccc None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:10.495290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94135c6a-6fe7-4bee-9343-9343e1410dc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:10.495994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94135c6a-6fe7-4bee-9343-9343e1410dc7 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:10.496217 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94135c6a-6fe7-4bee-9343-9343e1410dc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:10.496493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94135c6a-6fe7-4bee-9343-9343e1410dc7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:10.508618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94135c6a-6fe7-4bee-9343-9343e1410dc7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:10.510102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94135c6a-6fe7-4bee-9343-9343e1410dc7 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:10.521141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cadb53c-ece3-49ab-8242-b32d12827efb None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-94135c6a-6fe7-4bee-9343-9343e1410dc7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:10.526674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:10.527882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:10.529019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:10.529019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:10.529019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:10.530824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 373/707] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:10.531858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94135c6a-6fe7-4bee-9343-9343e1410dc7 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:10.571866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121e3fd7-f4a4-43fa-af3f-f56d8fa3608d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:10.572481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121e3fd7-f4a4-43fa-af3f-f56d8fa3608d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:10.572760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121e3fd7-f4a4-43fa-af3f-f56d8fa3608d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:10.573029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121e3fd7-f4a4-43fa-af3f-f56d8fa3608d None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:10.573725 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121e3fd7-f4a4-43fa-af3f-f56d8fa3608d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:10.574028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121e3fd7-f4a4-43fa-af3f-f56d8fa3608d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:10.575486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121e3fd7-f4a4-43fa-af3f-f56d8fa3608d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:10.588684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 335/708] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:10.589992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000c94b8-24d0-479f-a01b-6e6183f25c2e None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:10.590312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000c94b8-24d0-479f-a01b-6e6183f25c2e None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:10.590588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000c94b8-24d0-479f-a01b-6e6183f25c2e None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:10.594210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 336/709] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:10.594210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000c94b8-24d0-479f-a01b-6e6183f25c2e None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:10.597559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:10.653241 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94135c6a-6fe7-4bee-9343-9343e1410dc7 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:10.690986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4b501f-e089-4814-bbe5-3f61231edd66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:10.691934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4b501f-e089-4814-bbe5-3f61231edd66 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:10.692349 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4b501f-e089-4814-bbe5-3f61231edd66 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:10.692762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4b501f-e089-4814-bbe5-3f61231edd66 None admin] PATH_INFO: `/v3/services/network` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:10.693608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4b501f-e089-4814-bbe5-3f61231edd66 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:10.695274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4b501f-e089-4814-bbe5-3f61231edd66 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:10.700565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000c94b8-24d0-479f-a01b-6e6183f25c2e None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-bc4b501f-e089-4814-bbe5-3f61231edd66 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:10.707901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:10.708986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 03:27:10.709853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:10.714084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 374/710] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:10 2022] GET /identity/v3/services/network => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:10.714974 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4b501f-e089-4814-bbe5-3f61231edd66 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:10.717715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:10.739076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173362d5-6fb7-4db7-b9d1-323ceeac6a3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:10.739685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173362d5-6fb7-4db7-b9d1-323ceeac6a3e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:10.739969 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173362d5-6fb7-4db7-b9d1-323ceeac6a3e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:10.740232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173362d5-6fb7-4db7-b9d1-323ceeac6a3e None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:10.740867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173362d5-6fb7-4db7-b9d1-323ceeac6a3e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:10.741167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173362d5-6fb7-4db7-b9d1-323ceeac6a3e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:10.742579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173362d5-6fb7-4db7-b9d1-323ceeac6a3e None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:10.759295 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 337/711] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:10.760396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c690916-c3a5-47b2-9c8f-3235f177a534 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:10.760686 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c690916-c3a5-47b2-9c8f-3235f177a534 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:10.760965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c690916-c3a5-47b2-9c8f-3235f177a534 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:10.781098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1c690916-c3a5-47b2-9c8f-3235f177a534 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:10.782318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1c690916-c3a5-47b2-9c8f-3235f177a534 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:10.823777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4b501f-e089-4814-bbe5-3f61231edd66 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:10.842941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:10.852705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 338/712] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:27:10.855896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c690916-c3a5-47b2-9c8f-3235f177a534 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:10.874078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb3340a-1e51-4acf-8f8d-b3cf81cac8e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:10.874867 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb3340a-1e51-4acf-8f8d-b3cf81cac8e1 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:10.875696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb3340a-1e51-4acf-8f8d-b3cf81cac8e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:10.876081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb3340a-1e51-4acf-8f8d-b3cf81cac8e1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:10.876971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb3340a-1e51-4acf-8f8d-b3cf81cac8e1 None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=4591bec74a4641af818b2d75b657115e) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:10.877381 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb3340a-1e51-4acf-8f8d-b3cf81cac8e1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:10.879980 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb3340a-1e51-4acf-8f8d-b3cf81cac8e1 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:10.888286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 375/713] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:10 2022] GET /identity/v3/role_assignments?user.id=4591bec74a4641af818b2d75b657115e&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:10.889226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb3340a-1e51-4acf-8f8d-b3cf81cac8e1 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:10.963901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:10.968684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c690916-c3a5-47b2-9c8f-3235f177a534 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:11.003584 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb3340a-1e51-4acf-8f8d-b3cf81cac8e1 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:11.011258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65564dbe-eb5b-4c2d-afd9-09f3271bcb98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:11.011956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65564dbe-eb5b-4c2d-afd9-09f3271bcb98 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:11.012235 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65564dbe-eb5b-4c2d-afd9-09f3271bcb98 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:11.012557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65564dbe-eb5b-4c2d-afd9-09f3271bcb98 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:11.013669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65564dbe-eb5b-4c2d-afd9-09f3271bcb98 None admin] RBAC: Adding query filter params (name=network) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:11.013669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65564dbe-eb5b-4c2d-afd9-09f3271bcb98 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:11.015086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65564dbe-eb5b-4c2d-afd9-09f3271bcb98 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:11.020993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 339/714] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:10 2022] GET /identity/v3/services?name=network => generated 124 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:11.031449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65564dbe-eb5b-4c2d-afd9-09f3271bcb98 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:11.045376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672b6b4b-5f24-456c-aa5f-075761098717 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:11.046421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672b6b4b-5f24-456c-aa5f-075761098717 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:11.046931 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672b6b4b-5f24-456c-aa5f-075761098717 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:11.047431 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672b6b4b-5f24-456c-aa5f-075761098717 None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:11.048297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672b6b4b-5f24-456c-aa5f-075761098717 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:11.050547 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672b6b4b-5f24-456c-aa5f-075761098717 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:11.111422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:11.150805 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 376/715] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:27:10 2022] POST /identity/v3/users => generated 254 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:11.161749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65564dbe-eb5b-4c2d-afd9-09f3271bcb98 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:11.206733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293d9a65-f139-491f-98d1-a918ba69c0bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:11.207731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293d9a65-f139-491f-98d1-a918ba69c0bc None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:11.208210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293d9a65-f139-491f-98d1-a918ba69c0bc None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:11.208684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293d9a65-f139-491f-98d1-a918ba69c0bc None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:11.210582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293d9a65-f139-491f-98d1-a918ba69c0bc None admin] RBAC: Adding query filter params (type=network) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:11.211228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293d9a65-f139-491f-98d1-a918ba69c0bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:11.212940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293d9a65-f139-491f-98d1-a918ba69c0bc None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:11.226834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 340/716] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:11 2022] GET /identity/v3/services?type=network => generated 124 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:11.238725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672b6b4b-5f24-456c-aa5f-075761098717 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:11.368947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672b6b4b-5f24-456c-aa5f-075761098717 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:11.397929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:11.411383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0aa46e-d1b0-42f4-88be-c8765f7c03c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:11.412292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0aa46e-d1b0-42f4-88be-c8765f7c03c6 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:11.412292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0aa46e-d1b0-42f4-88be-c8765f7c03c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:11.416566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0aa46e-d1b0-42f4-88be-c8765f7c03c6 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:11.416566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0aa46e-d1b0-42f4-88be-c8765f7c03c6 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:11.416566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0aa46e-d1b0-42f4-88be-c8765f7c03c6 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-dc0aa46e-d1b0-42f4-88be-c8765f7c03c6 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:11.428113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:11.429444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 03:27:11.430341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:11.433115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 377/717] 10.222.0.65 () {56 vars in 1129 bytes} [Wed Aug 24 03:27:11 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:11.438758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293d9a65-f139-491f-98d1-a918ba69c0bc None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:11.558933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293d9a65-f139-491f-98d1-a918ba69c0bc None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:11.592858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:11.608746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819ba646-ab16-484b-a435-e9df51a5dbde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:11.609360 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819ba646-ab16-484b-a435-e9df51a5dbde None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:11.609703 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819ba646-ab16-484b-a435-e9df51a5dbde None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:11.609977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819ba646-ab16-484b-a435-e9df51a5dbde None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:11.612393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819ba646-ab16-484b-a435-e9df51a5dbde None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:11.612393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819ba646-ab16-484b-a435-e9df51a5dbde None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:11.612779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819ba646-ab16-484b-a435-e9df51a5dbde None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:11.618446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 341/718] 10.222.0.65 () {56 vars in 1124 bytes} [Wed Aug 24 03:27:11 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:11.624214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0aa46e-d1b0-42f4-88be-c8765f7c03c6 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:11.760486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0aa46e-d1b0-42f4-88be-c8765f7c03c6 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:11.797391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3a352f-9073-4cac-8b45-37712cd372e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:11.798030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3a352f-9073-4cac-8b45-37712cd372e3 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:11.798359 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3a352f-9073-4cac-8b45-37712cd372e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:11.798630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3a352f-9073-4cac-8b45-37712cd372e3 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:11.799264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3a352f-9073-4cac-8b45-37712cd372e3 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:11.799721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3a352f-9073-4cac-8b45-37712cd372e3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:11.801270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3a352f-9073-4cac-8b45-37712cd372e3 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:11.807430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 378/719] 10.222.0.65 () {56 vars in 1124 bytes} [Wed Aug 24 03:27:11 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:12.444884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3b5e20-39a4-41ff-85bc-fc3736a0c48c None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:12.445404 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3b5e20-39a4-41ff-85bc-fc3736a0c48c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:12.445483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3b5e20-39a4-41ff-85bc-fc3736a0c48c None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:12.447163 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 342/720] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:12.452789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf48b5c-e5d6-44ef-a51e-e348bc074f55 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:12.453208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf48b5c-e5d6-44ef-a51e-e348bc074f55 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:12.453398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf48b5c-e5d6-44ef-a51e-e348bc074f55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:12.480568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-dcf48b5c-e5d6-44ef-a51e-e348bc074f55 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:12.481770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-dcf48b5c-e5d6-44ef-a51e-e348bc074f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:12.542904 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:12.571976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 379/721] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:27:12.852682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3b5e20-39a4-41ff-85bc-fc3736a0c48c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:12.983122 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3b5e20-39a4-41ff-85bc-fc3736a0c48c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:13.029836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e08ad51-3560-4742-bce4-005152b3dda2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:13.030577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e08ad51-3560-4742-bce4-005152b3dda2 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:13.030857 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e08ad51-3560-4742-bce4-005152b3dda2 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:13.031132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e08ad51-3560-4742-bce4-005152b3dda2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:13.051060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e08ad51-3560-4742-bce4-005152b3dda2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-9e08ad51-3560-4742-bce4-005152b3dda2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:13.056238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:13.057161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:13.057995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 343/722] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:13.063496 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf48b5c-e5d6-44ef-a51e-e348bc074f55 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:13.212139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf48b5c-e5d6-44ef-a51e-e348bc074f55 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:13.253551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d551cf3-0d8b-441c-838d-2a783887948e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:13.254617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d551cf3-0d8b-441c-838d-2a783887948e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:13.254898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d551cf3-0d8b-441c-838d-2a783887948e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:13.255165 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d551cf3-0d8b-441c-838d-2a783887948e None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:13.256093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d551cf3-0d8b-441c-838d-2a783887948e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:13.256532 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d551cf3-0d8b-441c-838d-2a783887948e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:13.257905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d551cf3-0d8b-441c-838d-2a783887948e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:13.266682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 380/723] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:27:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:13.271846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e08ad51-3560-4742-bce4-005152b3dda2 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:13.345099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:13.434831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e08ad51-3560-4742-bce4-005152b3dda2 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:13.484788 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc62382-305d-4fcb-abee-32dd93f60985 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:13.485399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc62382-305d-4fcb-abee-32dd93f60985 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:13.485609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc62382-305d-4fcb-abee-32dd93f60985 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:13.485876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc62382-305d-4fcb-abee-32dd93f60985 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:13.502014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc62382-305d-4fcb-abee-32dd93f60985 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:13.503677 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc62382-305d-4fcb-abee-32dd93f60985 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-acc62382-305d-4fcb-abee-32dd93f60985 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:13.523196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:13.524264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:13.525804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:13.525804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:13.525804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:13.525804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 344/724] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:13.530405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d551cf3-0d8b-441c-838d-2a783887948e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:13.553439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c9f346-6b9c-4ae4-862f-b787d0ce5dba None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:13.553748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c9f346-6b9c-4ae4-862f-b787d0ce5dba None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:13.553990 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c9f346-6b9c-4ae4-862f-b787d0ce5dba None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:13.555479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 345/725] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:13.561923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cfbe91-d5cb-4ee0-b0c5-9de0b9c551b8 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:13.562650 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cfbe91-d5cb-4ee0-b0c5-9de0b9c551b8 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:13.563156 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cfbe91-d5cb-4ee0-b0c5-9de0b9c551b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:13.588679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-02cfbe91-d5cb-4ee0-b0c5-9de0b9c551b8 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:13.592601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-02cfbe91-d5cb-4ee0-b0c5-9de0b9c551b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:13.618046 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:13.697531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 346/726] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:27:13.699784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247634b7-aaff-4050-980a-96cbecb2724c None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:13.700410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247634b7-aaff-4050-980a-96cbecb2724c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:13.700900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247634b7-aaff-4050-980a-96cbecb2724c None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:13.703122 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 347/727] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:13.711475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b1d0d2-2009-4672-b07c-7f8999e138f5 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:13.712002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b1d0d2-2009-4672-b07c-7f8999e138f5 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:13.712499 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b1d0d2-2009-4672-b07c-7f8999e138f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:13.732271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d551cf3-0d8b-441c-838d-2a783887948e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:13.747212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-40b1d0d2-2009-4672-b07c-7f8999e138f5 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:13.748709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-40b1d0d2-2009-4672-b07c-7f8999e138f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:13.772006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e895c90-b1de-4d83-80a4-70ca3091e981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:13.772618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e895c90-b1de-4d83-80a4-70ca3091e981 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:13.773040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e895c90-b1de-4d83-80a4-70ca3091e981 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:13.773327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e895c90-b1de-4d83-80a4-70ca3091e981 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:13.774111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e895c90-b1de-4d83-80a4-70ca3091e981 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:13.774445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e895c90-b1de-4d83-80a4-70ca3091e981 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:13.775894 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e895c90-b1de-4d83-80a4-70ca3091e981 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:13.791103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 381/728] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:13.799219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e895c90-b1de-4d83-80a4-70ca3091e981 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:13.806559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:13.847746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 348/729] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:27:13.849446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca91a97-8857-43a4-b7fb-befbc0de8df8 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:13.849930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca91a97-8857-43a4-b7fb-befbc0de8df8 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:13.850426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca91a97-8857-43a4-b7fb-befbc0de8df8 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:13.852287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 349/730] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:13.857828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1234c563-fa40-4aee-ba0b-e8467494615f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:13.857828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1234c563-fa40-4aee-ba0b-e8467494615f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:13.857828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1234c563-fa40-4aee-ba0b-e8467494615f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:13.877151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1234c563-fa40-4aee-ba0b-e8467494615f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:13.878915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1234c563-fa40-4aee-ba0b-e8467494615f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:13.957161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 350/731] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 24 03:27:13.998092 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e895c90-b1de-4d83-80a4-70ca3091e981 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:13.999766 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:14.055904 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496ae06-21d8-4a4c-8aad-243e015c1223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:14.056772 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496ae06-21d8-4a4c-8aad-243e015c1223 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:14.057056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496ae06-21d8-4a4c-8aad-243e015c1223 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:14.057379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496ae06-21d8-4a4c-8aad-243e015c1223 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:14.057638 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1234c563-fa40-4aee-ba0b-e8467494615f None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:14.075404 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2496ae06-21d8-4a4c-8aad-243e015c1223 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:14.076878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2496ae06-21d8-4a4c-8aad-243e015c1223 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-2496ae06-21d8-4a4c-8aad-243e015c1223 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:14.099769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:14.100807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:14.101777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:14.101777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:14.101777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:14.101777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:14.101777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:14.101777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:14.101777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:14.101777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:14.101777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:14.101777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:14.101777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 382/732] 10.222.0.65 () {56 vars in 1093 bytes} [Wed Aug 24 03:27:13 2022] GET /identity/v3/users/nova => generated 82 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:14.107382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496ae06-21d8-4a4c-8aad-243e015c1223 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:14.161866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:14.178342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1234c563-fa40-4aee-ba0b-e8467494615f None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:14.222191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c34fadd-5446-427d-b73b-5bf8dac6bfeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:14.223236 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c34fadd-5446-427d-b73b-5bf8dac6bfeb None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:14.223733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c34fadd-5446-427d-b73b-5bf8dac6bfeb None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:14.224222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c34fadd-5446-427d-b73b-5bf8dac6bfeb None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:14.225078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c34fadd-5446-427d-b73b-5bf8dac6bfeb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:14.226930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c34fadd-5446-427d-b73b-5bf8dac6bfeb None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:14.273400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496ae06-21d8-4a4c-8aad-243e015c1223 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:14.308506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51249b86-ba2f-43ca-973c-39e0e7f1be60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:14.309185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51249b86-ba2f-43ca-973c-39e0e7f1be60 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:14.309465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51249b86-ba2f-43ca-973c-39e0e7f1be60 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:14.309742 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51249b86-ba2f-43ca-973c-39e0e7f1be60 None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:14.310503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51249b86-ba2f-43ca-973c-39e0e7f1be60 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:14.310811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51249b86-ba2f-43ca-973c-39e0e7f1be60 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:14.312310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51249b86-ba2f-43ca-973c-39e0e7f1be60 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:14.335956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 383/733] 10.222.0.65 () {56 vars in 1133 bytes} [Wed Aug 24 03:27:14 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:14.335956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51249b86-ba2f-43ca-973c-39e0e7f1be60 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:14.336269 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 351/734] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:27:14 2022] POST /identity/v3/services => generated 240 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:14.336269 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c34fadd-5446-427d-b73b-5bf8dac6bfeb None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:14.372595 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:14.490384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51249b86-ba2f-43ca-973c-39e0e7f1be60 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:14.509713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c34fadd-5446-427d-b73b-5bf8dac6bfeb None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:14.536905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf1c193-ed5b-4ac7-9679-3714ebd9dda0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:14.537558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf1c193-ed5b-4ac7-9679-3714ebd9dda0 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:14.537748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf1c193-ed5b-4ac7-9679-3714ebd9dda0 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:14.538023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf1c193-ed5b-4ac7-9679-3714ebd9dda0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:14.551909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf1c193-ed5b-4ac7-9679-3714ebd9dda0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:14.555240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f14c9c-f082-4188-bf46-5072f80d05be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:14.555880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f14c9c-f082-4188-bf46-5072f80d05be None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:14.556158 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f14c9c-f082-4188-bf46-5072f80d05be None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:14.556478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f14c9c-f082-4188-bf46-5072f80d05be None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-dcf1c193-ed5b-4ac7-9679-3714ebd9dda0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:14.559062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:14.560191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:14.561179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:14.561179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:14.561179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:14.561179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:14.561179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:14.561179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:14.561179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:14.561179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:14.561179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:14.561179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:14.561179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:14.561179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:14.563325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 384/735] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:27:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:14.564317 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf1c193-ed5b-4ac7-9679-3714ebd9dda0 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:14.576170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f14c9c-f082-4188-bf46-5072f80d05be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-45f14c9c-f082-4188-bf46-5072f80d05be None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:14.583149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:14.584455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 352/736] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:14.586020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f14c9c-f082-4188-bf46-5072f80d05be None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:14.594389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:14.672744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:14.695536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf1c193-ed5b-4ac7-9679-3714ebd9dda0 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:14.713799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f14c9c-f082-4188-bf46-5072f80d05be None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:14.734366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ddee80-62a4-4643-8a63-66636c064574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:14.735245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ddee80-62a4-4643-8a63-66636c064574 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:14.735671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ddee80-62a4-4643-8a63-66636c064574 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:14.736169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ddee80-62a4-4643-8a63-66636c064574 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:14.748224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ddee80-62a4-4643-8a63-66636c064574 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:14.749922 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ddee80-62a4-4643-8a63-66636c064574 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:14.759315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12084d7-b725-4568-8eff-6c24a2133a81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:14.760135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12084d7-b725-4568-8eff-6c24a2133a81 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:14.760602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12084d7-b725-4568-8eff-6c24a2133a81 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:14.760864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12084d7-b725-4568-8eff-6c24a2133a81 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:14.761565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12084d7-b725-4568-8eff-6c24a2133a81 None admin] RBAC: Adding query filter params (name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:14.761863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12084d7-b725-4568-8eff-6c24a2133a81 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:14.763286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12084d7-b725-4568-8eff-6c24a2133a81 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-e9ddee80-62a4-4643-8a63-66636c064574 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:14.769755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:14.770762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:14.776731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:14.776731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:14.776731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:14.776731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 385/737] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:14.776731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ddee80-62a4-4643-8a63-66636c064574 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:14.779723 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 353/738] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:27:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:14.781084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20637e11-1cdd-43e5-be0d-14fa9521297f None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:14.781368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20637e11-1cdd-43e5-be0d-14fa9521297f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:14.781641 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20637e11-1cdd-43e5-be0d-14fa9521297f None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:14.783279 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 354/739] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:14.786278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20637e11-1cdd-43e5-be0d-14fa9521297f None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:14.873233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:14.892513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ddee80-62a4-4643-8a63-66636c064574 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:14.912244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20637e11-1cdd-43e5-be0d-14fa9521297f None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:14.923322 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:14.929002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9891564e-2c00-4398-b4ae-5bbb0f459395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:14.929609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9891564e-2c00-4398-b4ae-5bbb0f459395 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:14.929940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9891564e-2c00-4398-b4ae-5bbb0f459395 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:14.930277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9891564e-2c00-4398-b4ae-5bbb0f459395 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:14.941307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9891564e-2c00-4398-b4ae-5bbb0f459395 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:14.941307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9891564e-2c00-4398-b4ae-5bbb0f459395 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:14.941307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9891564e-2c00-4398-b4ae-5bbb0f459395 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:14.946982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 386/740] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:27:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:14.947740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9891564e-2c00-4398-b4ae-5bbb0f459395 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:14.952601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28776d8-4e35-47f5-8b3a-bbcc82611779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:14.953210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28776d8-4e35-47f5-8b3a-bbcc82611779 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:14.953487 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28776d8-4e35-47f5-8b3a-bbcc82611779 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:14.953757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28776d8-4e35-47f5-8b3a-bbcc82611779 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:14.954577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28776d8-4e35-47f5-8b3a-bbcc82611779 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:14.954883 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28776d8-4e35-47f5-8b3a-bbcc82611779 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:14.956434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28776d8-4e35-47f5-8b3a-bbcc82611779 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:14.970598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 355/741] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:14.972525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d54319a-02b9-4517-9220-dd3c3ca2dd64 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:14.972817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d54319a-02b9-4517-9220-dd3c3ca2dd64 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:14.973088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d54319a-02b9-4517-9220-dd3c3ca2dd64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:14.992626 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1d54319a-02b9-4517-9220-dd3c3ca2dd64 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:14.993848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1d54319a-02b9-4517-9220-dd3c3ca2dd64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:15.056156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9891564e-2c00-4398-b4ae-5bbb0f459395 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:15.080683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 356/742] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:14 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 24 03:27:15.080683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d54319a-02b9-4517-9220-dd3c3ca2dd64 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:15.101587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317ccce4-d516-443a-9e1c-f386aab53ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:15.102256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317ccce4-d516-443a-9e1c-f386aab53ad6 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:15.102615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317ccce4-d516-443a-9e1c-f386aab53ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:15.103461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317ccce4-d516-443a-9e1c-f386aab53ad6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:15.118415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317ccce4-d516-443a-9e1c-f386aab53ad6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:15.120700 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317ccce4-d516-443a-9e1c-f386aab53ad6 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-317ccce4-d516-443a-9e1c-f386aab53ad6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:15.140574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:15.141515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:15.142439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:15.142439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:15.142439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:15.146431 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 387/743] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:15.147201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317ccce4-d516-443a-9e1c-f386aab53ad6 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:15.147951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:15.198538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:15.199955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d54319a-02b9-4517-9220-dd3c3ca2dd64 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:15.240803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db0eeb2-ca4c-4afb-adb7-b10ca142c6c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:15.241820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db0eeb2-ca4c-4afb-adb7-b10ca142c6c3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:15.242378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db0eeb2-ca4c-4afb-adb7-b10ca142c6c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:15.242880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db0eeb2-ca4c-4afb-adb7-b10ca142c6c3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:15.254665 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db0eeb2-ca4c-4afb-adb7-b10ca142c6c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:15.255077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317ccce4-d516-443a-9e1c-f386aab53ad6 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:15.257862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db0eeb2-ca4c-4afb-adb7-b10ca142c6c3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-5db0eeb2-ca4c-4afb-adb7-b10ca142c6c3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:15.275382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:15.276375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:15.277286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:15.277286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:15.277286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:15.279689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 357/744] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:15.280972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db0eeb2-ca4c-4afb-adb7-b10ca142c6c3 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:15.292797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73372cb-e415-4253-b760-de73addaf652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:15.293489 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73372cb-e415-4253-b760-de73addaf652 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:15.293778 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73372cb-e415-4253-b760-de73addaf652 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:15.294040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73372cb-e415-4253-b760-de73addaf652 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:15.300934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73372cb-e415-4253-b760-de73addaf652 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:15.302417 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73372cb-e415-4253-b760-de73addaf652 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-e73372cb-e415-4253-b760-de73addaf652 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:15.313702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:15.314602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:15.320912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:15.320912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:15.320912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:15.320912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:15.320912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:15.320912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:15.320912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:15.320912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:15.320912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 388/745] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:15.320912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73372cb-e415-4253-b760-de73addaf652 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:15.458050 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73372cb-e415-4253-b760-de73addaf652 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:15.462269 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db0eeb2-ca4c-4afb-adb7-b10ca142c6c3 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:15.465444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:15.472126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:15.494829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08802351-c007-41d1-a9ed-bd82df012ccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:15.495462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08802351-c007-41d1-a9ed-bd82df012ccc None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:15.495742 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08802351-c007-41d1-a9ed-bd82df012ccc None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:15.496017 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08802351-c007-41d1-a9ed-bd82df012ccc None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:15.496682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08802351-c007-41d1-a9ed-bd82df012ccc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:15.496987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08802351-c007-41d1-a9ed-bd82df012ccc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:15.498480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08802351-c007-41d1-a9ed-bd82df012ccc None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:15.500136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9256adc-d88f-4956-a57e-c49899f1cd2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:15.501133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9256adc-d88f-4956-a57e-c49899f1cd2c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:15.502174 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9256adc-d88f-4956-a57e-c49899f1cd2c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:15.503128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9256adc-d88f-4956-a57e-c49899f1cd2c None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:15.503950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9256adc-d88f-4956-a57e-c49899f1cd2c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:15.504417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9256adc-d88f-4956-a57e-c49899f1cd2c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:15.506338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9256adc-d88f-4956-a57e-c49899f1cd2c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:15.521682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 389/746] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:15.521682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08802351-c007-41d1-a9ed-bd82df012ccc None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:15.524720 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 358/747] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:15.525508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9256adc-d88f-4956-a57e-c49899f1cd2c None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:15.625885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08802351-c007-41d1-a9ed-bd82df012ccc None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:15.641491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9256adc-d88f-4956-a57e-c49899f1cd2c None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:15.662072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8186ea7-6d14-4fff-8ea7-3d3fff11d38d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:15.662774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8186ea7-6d14-4fff-8ea7-3d3fff11d38d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:15.663567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8186ea7-6d14-4fff-8ea7-3d3fff11d38d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:15.663942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8186ea7-6d14-4fff-8ea7-3d3fff11d38d None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:15.664732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8186ea7-6d14-4fff-8ea7-3d3fff11d38d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:15.665135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8186ea7-6d14-4fff-8ea7-3d3fff11d38d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:15.666758 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8186ea7-6d14-4fff-8ea7-3d3fff11d38d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:15.678547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a4f5b4-7821-4e82-9317-e30333c7ea12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:15.683975 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 390/748] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:15.684143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a4f5b4-7821-4e82-9317-e30333c7ea12 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:15.684143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a4f5b4-7821-4e82-9317-e30333c7ea12 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:15.684143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a4f5b4-7821-4e82-9317-e30333c7ea12 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:15.684375 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8186ea7-6d14-4fff-8ea7-3d3fff11d38d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:15.684459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a4f5b4-7821-4e82-9317-e30333c7ea12 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:15.684459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a4f5b4-7821-4e82-9317-e30333c7ea12 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:15.711238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:15.750660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 359/749] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:27:15 2022] POST /identity/v3/services => generated 251 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:15.751687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a4f5b4-7821-4e82-9317-e30333c7ea12 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:15.790079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:15.814142 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8186ea7-6d14-4fff-8ea7-3d3fff11d38d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:15.882674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a4f5b4-7821-4e82-9317-e30333c7ea12 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:15.882912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78da336-10eb-4c27-8c41-ff6c0618db9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:15.883719 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78da336-10eb-4c27-8c41-ff6c0618db9e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:15.884072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78da336-10eb-4c27-8c41-ff6c0618db9e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:15.884417 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78da336-10eb-4c27-8c41-ff6c0618db9e None admin] PATH_INFO: `/v3/users/swift` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:15.896425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78da336-10eb-4c27-8c41-ff6c0618db9e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:15.898434 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78da336-10eb-4c27-8c41-ff6c0618db9e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-c78da336-10eb-4c27-8c41-ff6c0618db9e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:15.914458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:15.915424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:15.916378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:15.916378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:15.916378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:15.916378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:15.916378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:15.916378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:15.916378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:15.916378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:15.916378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 24 03:27:15.916378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:15.916378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 391/750] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:15 2022] GET /identity/v3/users/swift => generated 83 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:15.916965 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78da336-10eb-4c27-8c41-ff6c0618db9e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:15.931479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75c8c6b-97c3-4ce3-893d-afbec788d951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:15.932440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75c8c6b-97c3-4ce3-893d-afbec788d951 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:15.932725 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75c8c6b-97c3-4ce3-893d-afbec788d951 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:15.933294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75c8c6b-97c3-4ce3-893d-afbec788d951 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:15.944575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75c8c6b-97c3-4ce3-893d-afbec788d951 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:15.946022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75c8c6b-97c3-4ce3-893d-afbec788d951 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-e75c8c6b-97c3-4ce3-893d-afbec788d951 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:15.960496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:15.961485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:15.962490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:15.962490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:15.962490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:15.962490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:15.962490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:15.962490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:15.962490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:15.962490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:15.962490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 03:27:15.962490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:15.962490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 360/751] 10.222.0.65 () {56 vars in 1117 bytes} [Wed Aug 24 03:27:15 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:15.963099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75c8c6b-97c3-4ce3-893d-afbec788d951 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:16.023635 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:16.044340 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78da336-10eb-4c27-8c41-ff6c0618db9e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:16.061184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:16.074709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75c8c6b-97c3-4ce3-893d-afbec788d951 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:16.084780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e5e8e7-bb70-4814-bbc4-263df3d6eb24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:16.086140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e5e8e7-bb70-4814-bbc4-263df3d6eb24 None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:16.086632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e5e8e7-bb70-4814-bbc4-263df3d6eb24 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:16.087398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e5e8e7-bb70-4814-bbc4-263df3d6eb24 None admin] PATH_INFO: `/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/e5896018712d4b82a781365e478ee855/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:16.090033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e5e8e7-bb70-4814-bbc4-263df3d6eb24 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c653d5fa4304f298a295ce635efd5e6, user_id=e5896018712d4b82a781365e478ee855, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:16.092797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e5e8e7-bb70-4814-bbc4-263df3d6eb24 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:16.112069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d030de-67b2-4c9c-8be0-3398231afa20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:16.112714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d030de-67b2-4c9c-8be0-3398231afa20 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:16.112955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d030de-67b2-4c9c-8be0-3398231afa20 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:16.113225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d030de-67b2-4c9c-8be0-3398231afa20 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:16.113987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d030de-67b2-4c9c-8be0-3398231afa20 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:16.114292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d030de-67b2-4c9c-8be0-3398231afa20 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:16.115686 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d030de-67b2-4c9c-8be0-3398231afa20 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:16.138703 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 361/752] 10.222.0.65 () {56 vars in 1135 bytes} [Wed Aug 24 03:27:15 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:16.139860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d030de-67b2-4c9c-8be0-3398231afa20 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:16.179888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 392/753] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:27:15 2022] PUT /identity/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/e5896018712d4b82a781365e478ee855/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:27:16.187472 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e5e8e7-bb70-4814-bbc4-263df3d6eb24 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:16.278201 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d030de-67b2-4c9c-8be0-3398231afa20 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:16.317686 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e00943-f553-418a-86a3-7940732ae4b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:16.318361 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e00943-f553-418a-86a3-7940732ae4b9 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:16.318642 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e00943-f553-418a-86a3-7940732ae4b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:16.318915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e00943-f553-418a-86a3-7940732ae4b9 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:16.319654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e00943-f553-418a-86a3-7940732ae4b9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:16.319959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e00943-f553-418a-86a3-7940732ae4b9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:16.321412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e00943-f553-418a-86a3-7940732ae4b9 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:16.345354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 362/754] 10.222.0.65 () {56 vars in 1149 bytes} [Wed Aug 24 03:27:16 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:16.352127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e00943-f553-418a-86a3-7940732ae4b9 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:16.356636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:16.403287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e5e8e7-bb70-4814-bbc4-263df3d6eb24 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:16.444952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0dd47c7-4bfc-4f49-a993-54b3dbd0fa73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:16.445482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dd47c7-4bfc-4f49-a993-54b3dbd0fa73 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:16.445758 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dd47c7-4bfc-4f49-a993-54b3dbd0fa73 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:16.446025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dd47c7-4bfc-4f49-a993-54b3dbd0fa73 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:16.461941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:16.493298 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e00943-f553-418a-86a3-7940732ae4b9 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:16.534374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf5147f-ee9b-40f4-8f8e-b4591b464721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:16.535231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf5147f-ee9b-40f4-8f8e-b4591b464721 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:16.535431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf5147f-ee9b-40f4-8f8e-b4591b464721 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:16.535703 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf5147f-ee9b-40f4-8f8e-b4591b464721 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:16.536471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0dd47c7-4bfc-4f49-a993-54b3dbd0fa73 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:16.538205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0dd47c7-4bfc-4f49-a993-54b3dbd0fa73 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:16.606518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf5147f-ee9b-40f4-8f8e-b4591b464721 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:16.608235 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf5147f-ee9b-40f4-8f8e-b4591b464721 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:16.611002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:16.649410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 393/755] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:16 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:27:16.651476 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2331b523-e034-4c8e-834e-2a86f1cf8744 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:16.651973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2331b523-e034-4c8e-834e-2a86f1cf8744 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:16.652439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2331b523-e034-4c8e-834e-2a86f1cf8744 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:16.654126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 394/756] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:16.659737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f55d04-d794-4158-a639-fcb73d57259e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:16.660302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f55d04-d794-4158-a639-fcb73d57259e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:16.660920 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f55d04-d794-4158-a639-fcb73d57259e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:16.680486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-84f55d04-d794-4158-a639-fcb73d57259e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:16.682089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-84f55d04-d794-4158-a639-fcb73d57259e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:16.718308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 363/757] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:16 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 24 03:27:16.719480 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf5147f-ee9b-40f4-8f8e-b4591b464721 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:16.726593 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:16.774250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 395/758] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:16.775144 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f55d04-d794-4158-a639-fcb73d57259e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:16.875243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf5147f-ee9b-40f4-8f8e-b4591b464721 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:16.879035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:16.948475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ccb377-606c-420d-b476-44eb91f2573c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:16.949137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ccb377-606c-420d-b476-44eb91f2573c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:16.949415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ccb377-606c-420d-b476-44eb91f2573c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:16.949690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ccb377-606c-420d-b476-44eb91f2573c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:16.969205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ccb377-606c-420d-b476-44eb91f2573c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:16.970619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ccb377-606c-420d-b476-44eb91f2573c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:17.005922 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f55d04-d794-4158-a639-fcb73d57259e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-75ccb377-606c-420d-b476-44eb91f2573c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:17.006173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:17.007886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:17.009144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:17.009144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:17.009144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:17.009144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 364/759] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:17.021142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ccb377-606c-420d-b476-44eb91f2573c None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:17.123075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef96109-5355-414e-b2b6-1b9ddeece953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:17.124438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef96109-5355-414e-b2b6-1b9ddeece953 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:17.125155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef96109-5355-414e-b2b6-1b9ddeece953 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:17.125857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef96109-5355-414e-b2b6-1b9ddeece953 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:17.146436 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef96109-5355-414e-b2b6-1b9ddeece953 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:17.149255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef96109-5355-414e-b2b6-1b9ddeece953 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-aef96109-5355-414e-b2b6-1b9ddeece953 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:17.176961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:17.178813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:17.180570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:17.180570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:17.180570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:17.184219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 396/760] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 409 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:17.190847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef96109-5355-414e-b2b6-1b9ddeece953 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:17.238573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:17.266394 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ccb377-606c-420d-b476-44eb91f2573c None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:17.324354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8660fd05-39ba-4815-b5c4-2b04c5b538d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:17.325560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8660fd05-39ba-4815-b5c4-2b04c5b538d2 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:17.326145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8660fd05-39ba-4815-b5c4-2b04c5b538d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:17.326818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8660fd05-39ba-4815-b5c4-2b04c5b538d2 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:17.328006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8660fd05-39ba-4815-b5c4-2b04c5b538d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:17.328592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8660fd05-39ba-4815-b5c4-2b04c5b538d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:17.331449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8660fd05-39ba-4815-b5c4-2b04c5b538d2 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:17.354504 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 365/761] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:17.355666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8660fd05-39ba-4815-b5c4-2b04c5b538d2 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:17.369854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef96109-5355-414e-b2b6-1b9ddeece953 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:17.375794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:17.418603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e31845d-39b9-405f-ba9d-a441d26ab3bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:17.419690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e31845d-39b9-405f-ba9d-a441d26ab3bb None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:17.420020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e31845d-39b9-405f-ba9d-a441d26ab3bb None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:17.420303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e31845d-39b9-405f-ba9d-a441d26ab3bb None admin] PATH_INFO: `/v3/services/network` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:17.420903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e31845d-39b9-405f-ba9d-a441d26ab3bb None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:17.422405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e31845d-39b9-405f-ba9d-a441d26ab3bb None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-5e31845d-39b9-405f-ba9d-a441d26ab3bb None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:17.436429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:17.437473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:17.443656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 397/762] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:17 2022] GET /identity/v3/services/network => generated 88 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:17.444518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e31845d-39b9-405f-ba9d-a441d26ab3bb None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:17.484720 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8660fd05-39ba-4815-b5c4-2b04c5b538d2 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:17.522410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db92e66-d882-4371-9f7d-5927f3732bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:17.523336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db92e66-d882-4371-9f7d-5927f3732bed None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:17.523817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db92e66-d882-4371-9f7d-5927f3732bed None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:17.524304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db92e66-d882-4371-9f7d-5927f3732bed None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:17.525137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db92e66-d882-4371-9f7d-5927f3732bed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:17.525730 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db92e66-d882-4371-9f7d-5927f3732bed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:17.527364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db92e66-d882-4371-9f7d-5927f3732bed None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:17.540270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 366/763] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:17.541914 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db92e66-d882-4371-9f7d-5927f3732bed None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:17.566254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:17.580057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e31845d-39b9-405f-ba9d-a441d26ab3bb None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:17.618281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681805c0-69c3-43a4-b37a-0ffd416b3f92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:17.618913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681805c0-69c3-43a4-b37a-0ffd416b3f92 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:17.619192 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681805c0-69c3-43a4-b37a-0ffd416b3f92 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:17.619465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681805c0-69c3-43a4-b37a-0ffd416b3f92 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:17.624064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:17.627659 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681805c0-69c3-43a4-b37a-0ffd416b3f92 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:17.629360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681805c0-69c3-43a4-b37a-0ffd416b3f92 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-681805c0-69c3-43a4-b37a-0ffd416b3f92 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:17.642094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:17.643054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:17.644006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:17.644006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:17.644006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:17.644006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:17.644006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:17.644006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:17.644006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:17.644006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:17.650889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db92e66-d882-4371-9f7d-5927f3732bed None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:17.651117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 398/764] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:17.652457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681805c0-69c3-43a4-b37a-0ffd416b3f92 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:17.687144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cffc498-a39e-427f-8f89-112f1ee7302a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:17.687904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cffc498-a39e-427f-8f89-112f1ee7302a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:17.688287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cffc498-a39e-427f-8f89-112f1ee7302a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:17.688658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cffc498-a39e-427f-8f89-112f1ee7302a None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:17.689395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cffc498-a39e-427f-8f89-112f1ee7302a None admin] RBAC: Adding query filter params (name=network) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:17.689803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cffc498-a39e-427f-8f89-112f1ee7302a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:17.692462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cffc498-a39e-427f-8f89-112f1ee7302a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:17.701300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 367/765] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:17 2022] GET /identity/v3/services?name=network => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:17.702376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cffc498-a39e-427f-8f89-112f1ee7302a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:17.763251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681805c0-69c3-43a4-b37a-0ffd416b3f92 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:17.800028 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df45b8c3-a4fb-4def-9b02-a010743abe59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:17.801042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df45b8c3-a4fb-4def-9b02-a010743abe59 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:17.801593 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df45b8c3-a4fb-4def-9b02-a010743abe59 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:17.802043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df45b8c3-a4fb-4def-9b02-a010743abe59 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:17.809186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cffc498-a39e-427f-8f89-112f1ee7302a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:17.809618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df45b8c3-a4fb-4def-9b02-a010743abe59 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:17.811267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df45b8c3-a4fb-4def-9b02-a010743abe59 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-df45b8c3-a4fb-4def-9b02-a010743abe59 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:17.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:17.824514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:17.825413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:17.825413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:17.825413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:17.825413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:17.825413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:17.825413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:17.825413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:17.825413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:17.832517 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 399/766] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:17.833378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df45b8c3-a4fb-4def-9b02-a010743abe59 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:17.844547 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:17.845636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0613617-db40-40d7-bc63-0a3446a6ec89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:17.846341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0613617-db40-40d7-bc63-0a3446a6ec89 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:17.846640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0613617-db40-40d7-bc63-0a3446a6ec89 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:17.846907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0613617-db40-40d7-bc63-0a3446a6ec89 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:17.847599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0613617-db40-40d7-bc63-0a3446a6ec89 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:17.847902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0613617-db40-40d7-bc63-0a3446a6ec89 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:17.849324 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0613617-db40-40d7-bc63-0a3446a6ec89 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:17.859169 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:17.868446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 368/767] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:17.869576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e22d3ba-1730-40e8-a0ea-5aa9d031e1ed None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:17.869860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e22d3ba-1730-40e8-a0ea-5aa9d031e1ed None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:17.870137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e22d3ba-1730-40e8-a0ea-5aa9d031e1ed None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:17.871567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 369/768] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:17.872076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e22d3ba-1730-40e8-a0ea-5aa9d031e1ed None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:17.945096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df45b8c3-a4fb-4def-9b02-a010743abe59 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:17.980043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e22d3ba-1730-40e8-a0ea-5aa9d031e1ed None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:17.981211 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5039ef-8ef5-447e-a171-2fdabaf0d635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:17.981981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5039ef-8ef5-447e-a171-2fdabaf0d635 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:17.983077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5039ef-8ef5-447e-a171-2fdabaf0d635 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:17.983341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5039ef-8ef5-447e-a171-2fdabaf0d635 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:17.983973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5039ef-8ef5-447e-a171-2fdabaf0d635 None admin] RBAC: Adding query filter params (type=network) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:17.984273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5039ef-8ef5-447e-a171-2fdabaf0d635 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:17.985645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5039ef-8ef5-447e-a171-2fdabaf0d635 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:17.997004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 400/769] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:17 2022] GET /identity/v3/services?type=network => generated 350 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:17.997004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5039ef-8ef5-447e-a171-2fdabaf0d635 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:18.018953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908458c5-2c60-4bb7-8262-bd1d708fb568 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:18.019657 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908458c5-2c60-4bb7-8262-bd1d708fb568 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:18.020004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908458c5-2c60-4bb7-8262-bd1d708fb568 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:18.020347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908458c5-2c60-4bb7-8262-bd1d708fb568 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:18.021110 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908458c5-2c60-4bb7-8262-bd1d708fb568 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:18.021417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908458c5-2c60-4bb7-8262-bd1d708fb568 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:18.022859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908458c5-2c60-4bb7-8262-bd1d708fb568 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:18.035133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 370/770] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:18.036692 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0488c782-8677-4e67-a708-38314edb804d None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:18.037122 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0488c782-8677-4e67-a708-38314edb804d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:18.038028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0488c782-8677-4e67-a708-38314edb804d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:18.058455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-0488c782-8677-4e67-a708-38314edb804d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:18.059835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0488c782-8677-4e67-a708-38314edb804d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:18.070641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:18.101300 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5039ef-8ef5-447e-a171-2fdabaf0d635 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:18.128672 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:18.137981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddc9fbd-23cf-412d-98f7-dec94bbd7e9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:18.138668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddc9fbd-23cf-412d-98f7-dec94bbd7e9b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:18.138951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddc9fbd-23cf-412d-98f7-dec94bbd7e9b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:18.139224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddc9fbd-23cf-412d-98f7-dec94bbd7e9b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:18.139946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddc9fbd-23cf-412d-98f7-dec94bbd7e9b None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=4591bec74a4641af818b2d75b657115e) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:18.150234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddc9fbd-23cf-412d-98f7-dec94bbd7e9b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:18.150392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 371/771] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:18 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:18.150392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0488c782-8677-4e67-a708-38314edb804d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:18.150581 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddc9fbd-23cf-412d-98f7-dec94bbd7e9b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:18.155656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 401/772] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:17 2022] GET /identity/v3/role_assignments?user.id=4591bec74a4641af818b2d75b657115e&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=43ca25467bc6484da3a50508efb79230 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:18.156618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddc9fbd-23cf-412d-98f7-dec94bbd7e9b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:18.274234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0488c782-8677-4e67-a708-38314edb804d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:18.296444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddc9fbd-23cf-412d-98f7-dec94bbd7e9b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:18.315505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a5330a-b4a3-411e-92f4-cd8778a0f6b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:18.316401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a5330a-b4a3-411e-92f4-cd8778a0f6b4 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:18.316751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a5330a-b4a3-411e-92f4-cd8778a0f6b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:18.317137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a5330a-b4a3-411e-92f4-cd8778a0f6b4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:18.317932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a5330a-b4a3-411e-92f4-cd8778a0f6b4 None admin] RBAC: Adding query filter params (interface=public, service_id=cc57661148fe461bb385353082597d71, region_id=RegionOne) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:18.318330 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a5330a-b4a3-411e-92f4-cd8778a0f6b4 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:18.320076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a5330a-b4a3-411e-92f4-cd8778a0f6b4 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:18.336315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 372/773] 10.222.0.65 () {56 vars in 1251 bytes} [Wed Aug 24 03:27:18 2022] GET /identity/v3/endpoints?service_id=cc57661148fe461bb385353082597d71&interface=public®ion_id=RegionOne => generated 194 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:18.337849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe8f2f2-a2da-4dd8-bf32-394c098f149c None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:18.338082 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe8f2f2-a2da-4dd8-bf32-394c098f149c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:18.338407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe8f2f2-a2da-4dd8-bf32-394c098f149c None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:18.339733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 373/774] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:18.341235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eee77da-049d-4f28-82c4-178d6efebb62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:18.341885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe8f2f2-a2da-4dd8-bf32-394c098f149c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:18.342234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eee77da-049d-4f28-82c4-178d6efebb62 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:18.342673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eee77da-049d-4f28-82c4-178d6efebb62 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:18.343093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eee77da-049d-4f28-82c4-178d6efebb62 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:18.344041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eee77da-049d-4f28-82c4-178d6efebb62 None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=c63ade06fed946e79dc1e4b5ecfab5e6) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:18.344495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eee77da-049d-4f28-82c4-178d6efebb62 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:18.346101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eee77da-049d-4f28-82c4-178d6efebb62 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:18.350187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:18.354195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 402/775] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:18 2022] GET /identity/v3/role_assignments?user.id=c63ade06fed946e79dc1e4b5ecfab5e6&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:18.355664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fcd4cd-44ac-4a7b-be09-7b3083feb7a6 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:18.356103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fcd4cd-44ac-4a7b-be09-7b3083feb7a6 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:18.356524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fcd4cd-44ac-4a7b-be09-7b3083feb7a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:18.376575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-24fcd4cd-44ac-4a7b-be09-7b3083feb7a6 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:18.378235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-24fcd4cd-44ac-4a7b-be09-7b3083feb7a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:18.434954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:18.476023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 403/776] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:18 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:18.504665 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe8f2f2-a2da-4dd8-bf32-394c098f149c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:18.538989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fcd4cd-44ac-4a7b-be09-7b3083feb7a6 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:18.564862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cced4662-6b8d-4d46-a5b7-75cb96431880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:18.567001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cced4662-6b8d-4d46-a5b7-75cb96431880 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:18.567500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cced4662-6b8d-4d46-a5b7-75cb96431880 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:18.567969 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cced4662-6b8d-4d46-a5b7-75cb96431880 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:18.568876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cced4662-6b8d-4d46-a5b7-75cb96431880 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:18.570584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cced4662-6b8d-4d46-a5b7-75cb96431880 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:18.577142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 374/777] 10.222.0.65 () {56 vars in 1087 bytes} [Wed Aug 24 03:27:18 2022] GET /identity/v3/services => generated 772 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:18.672020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fcd4cd-44ac-4a7b-be09-7b3083feb7a6 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:18.707481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:18.722258 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e390e66-cf0b-4fe5-a063-d55a4a47290a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:18.728247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e390e66-cf0b-4fe5-a063-d55a4a47290a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:18.728247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e390e66-cf0b-4fe5-a063-d55a4a47290a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:18.728247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e390e66-cf0b-4fe5-a063-d55a4a47290a None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:18.728247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e390e66-cf0b-4fe5-a063-d55a4a47290a None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:18.728247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e390e66-cf0b-4fe5-a063-d55a4a47290a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-0e390e66-cf0b-4fe5-a063-d55a4a47290a None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:18.740495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:18.741539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:18.742413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 404/778] 10.222.0.65 () {56 vars in 1129 bytes} [Wed Aug 24 03:27:18 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:18.750327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cced4662-6b8d-4d46-a5b7-75cb96431880 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:18.773825 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e390e66-cf0b-4fe5-a063-d55a4a47290a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:18.889675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e390e66-cf0b-4fe5-a063-d55a4a47290a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:18.926559 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05caf080-3982-4a0c-9e76-d6cf35807dc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:18.927195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05caf080-3982-4a0c-9e76-d6cf35807dc4 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:18.927469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05caf080-3982-4a0c-9e76-d6cf35807dc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:18.927737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05caf080-3982-4a0c-9e76-d6cf35807dc4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:18.938665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05caf080-3982-4a0c-9e76-d6cf35807dc4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-05caf080-3982-4a0c-9e76-d6cf35807dc4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:18.944587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:18.945608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:18.946484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 405/779] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:18.952724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05caf080-3982-4a0c-9e76-d6cf35807dc4 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:18.982395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cced4662-6b8d-4d46-a5b7-75cb96431880 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:19.017571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:19.050697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a086107c-9f19-43d7-8604-f3a76154595a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:19.051700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a086107c-9f19-43d7-8604-f3a76154595a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:19.052275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a086107c-9f19-43d7-8604-f3a76154595a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:19.052953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a086107c-9f19-43d7-8604-f3a76154595a None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:19.054801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a086107c-9f19-43d7-8604-f3a76154595a None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:19.055314 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a086107c-9f19-43d7-8604-f3a76154595a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:19.057075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a086107c-9f19-43d7-8604-f3a76154595a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:19.058752 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:19.065859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 375/780] 10.222.0.65 () {56 vars in 1124 bytes} [Wed Aug 24 03:27:18 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:19.072068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a086107c-9f19-43d7-8604-f3a76154595a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:19.149242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05caf080-3982-4a0c-9e76-d6cf35807dc4 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:19.190584 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda893ed-c9dd-49da-8a79-323ef54654cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:19.191570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda893ed-c9dd-49da-8a79-323ef54654cb None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:19.192128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda893ed-c9dd-49da-8a79-323ef54654cb None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:19.192601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda893ed-c9dd-49da-8a79-323ef54654cb None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:19.193462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda893ed-c9dd-49da-8a79-323ef54654cb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:19.193961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda893ed-c9dd-49da-8a79-323ef54654cb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:19.195640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda893ed-c9dd-49da-8a79-323ef54654cb None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:19.204512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 406/781] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:27:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:19.210643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda893ed-c9dd-49da-8a79-323ef54654cb None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:19.241237 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a086107c-9f19-43d7-8604-f3a76154595a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:19.280901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d989cf-c3fb-4ef6-b7c8-a5bf590018c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:19.281787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d989cf-c3fb-4ef6-b7c8-a5bf590018c9 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:19.282067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d989cf-c3fb-4ef6-b7c8-a5bf590018c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:19.282360 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d989cf-c3fb-4ef6-b7c8-a5bf590018c9 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:19.282991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d989cf-c3fb-4ef6-b7c8-a5bf590018c9 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:19.283290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d989cf-c3fb-4ef6-b7c8-a5bf590018c9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:19.286072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d989cf-c3fb-4ef6-b7c8-a5bf590018c9 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:19.292339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 376/782] 10.222.0.65 () {56 vars in 1124 bytes} [Wed Aug 24 03:27:19 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:19.307428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d989cf-c3fb-4ef6-b7c8-a5bf590018c9 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:19.389899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda893ed-c9dd-49da-8a79-323ef54654cb None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:19.419197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:19.423406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:19.449345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ba3f51-0b22-49f8-ac9c-dd1ee89645bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:19.450244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ba3f51-0b22-49f8-ac9c-dd1ee89645bf None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:19.451797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ba3f51-0b22-49f8-ac9c-dd1ee89645bf None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:19.452223 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ba3f51-0b22-49f8-ac9c-dd1ee89645bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:19.479251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ba3f51-0b22-49f8-ac9c-dd1ee89645bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:19.480914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ba3f51-0b22-49f8-ac9c-dd1ee89645bf None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-c4ba3f51-0b22-49f8-ac9c-dd1ee89645bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:19.503750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:19.504938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:19.506316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:19.506316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:19.506316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:19.518611 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 407/783] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:19.524353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d989cf-c3fb-4ef6-b7c8-a5bf590018c9 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:19.538521 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ba3f51-0b22-49f8-ac9c-dd1ee89645bf None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:19.570984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970317f2-58e3-402e-85a9-6deb159e5584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:19.571615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970317f2-58e3-402e-85a9-6deb159e5584 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:19.571886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970317f2-58e3-402e-85a9-6deb159e5584 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:19.572134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970317f2-58e3-402e-85a9-6deb159e5584 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:19.572834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970317f2-58e3-402e-85a9-6deb159e5584 None admin] RBAC: Adding query filter params (interface=public, service_id=830a29b620f940c6a642cb63e1783b3e, region_id=RegionOne) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:19.573136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970317f2-58e3-402e-85a9-6deb159e5584 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:19.574517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970317f2-58e3-402e-85a9-6deb159e5584 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:19.589618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 377/784] 10.222.0.65 () {56 vars in 1251 bytes} [Wed Aug 24 03:27:19 2022] GET /identity/v3/endpoints?service_id=830a29b620f940c6a642cb63e1783b3e&interface=public®ion_id=RegionOne => generated 194 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:19.598032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970317f2-58e3-402e-85a9-6deb159e5584 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:19.716885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ba3f51-0b22-49f8-ac9c-dd1ee89645bf None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:19.756090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb778768-02c1-4fc9-806a-3c24a1f3e169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:19.756702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb778768-02c1-4fc9-806a-3c24a1f3e169 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:19.757870 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb778768-02c1-4fc9-806a-3c24a1f3e169 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:19.758179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb778768-02c1-4fc9-806a-3c24a1f3e169 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:19.758810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb778768-02c1-4fc9-806a-3c24a1f3e169 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:19.759108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb778768-02c1-4fc9-806a-3c24a1f3e169 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:19.760485 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb778768-02c1-4fc9-806a-3c24a1f3e169 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:19.779603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 408/785] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:19.787038 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb778768-02c1-4fc9-806a-3c24a1f3e169 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:19.796103 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970317f2-58e3-402e-85a9-6deb159e5584 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:19.819288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:19.824014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:19.891620 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0358af-0d66-406b-9893-4215286e5664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:19.892241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0358af-0d66-406b-9893-4215286e5664 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:19.892517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0358af-0d66-406b-9893-4215286e5664 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:19.892784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0358af-0d66-406b-9893-4215286e5664 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:19.893469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0358af-0d66-406b-9893-4215286e5664 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:19.895101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0358af-0d66-406b-9893-4215286e5664 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:19.906047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 378/786] 10.222.0.65 () {56 vars in 1087 bytes} [Wed Aug 24 03:27:19 2022] GET /identity/v3/services => generated 772 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:19.912726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb778768-02c1-4fc9-806a-3c24a1f3e169 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:19.952025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadceb8b-e63c-49f8-b833-27b73ede30f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:19.952694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadceb8b-e63c-49f8-b833-27b73ede30f1 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:19.952974 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadceb8b-e63c-49f8-b833-27b73ede30f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:19.953243 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadceb8b-e63c-49f8-b833-27b73ede30f1 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:19.961664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadceb8b-e63c-49f8-b833-27b73ede30f1 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:19.963153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadceb8b-e63c-49f8-b833-27b73ede30f1 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-fadceb8b-e63c-49f8-b833-27b73ede30f1 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:19.976302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:19.977286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:19.978266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:19.978266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:19.978266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:19.978266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:19.978266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:19.978266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:19.978266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:19.978266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:19.978266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:19.978266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:19.978266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 409/787] 10.222.0.65 () {56 vars in 1093 bytes} [Wed Aug 24 03:27:19 2022] GET /identity/v3/users/nova => generated 82 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:19.983221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0358af-0d66-406b-9893-4215286e5664 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:20.126568 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0358af-0d66-406b-9893-4215286e5664 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:20.163691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23fc52b-c708-41ef-a312-64513fbcccf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:20.164356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23fc52b-c708-41ef-a312-64513fbcccf3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:20.164634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23fc52b-c708-41ef-a312-64513fbcccf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:20.164904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23fc52b-c708-41ef-a312-64513fbcccf3 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:20.165676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23fc52b-c708-41ef-a312-64513fbcccf3 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:20.165982 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23fc52b-c708-41ef-a312-64513fbcccf3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:20.167498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23fc52b-c708-41ef-a312-64513fbcccf3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:20.188247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 379/788] 10.222.0.65 () {56 vars in 1133 bytes} [Wed Aug 24 03:27:19 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:20.193937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadceb8b-e63c-49f8-b833-27b73ede30f1 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:20.290115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:20.293550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ffd5bf-2c90-4a8a-b81e-fe7299507d9d None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:20.293836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ffd5bf-2c90-4a8a-b81e-fe7299507d9d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:20.294109 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ffd5bf-2c90-4a8a-b81e-fe7299507d9d None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:20.295496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 380/789] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:20.301462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de18b71c-cd53-4ad1-aa58-62e263d9bf75 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:20.301744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de18b71c-cd53-4ad1-aa58-62e263d9bf75 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:20.302021 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de18b71c-cd53-4ad1-aa58-62e263d9bf75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:20.310954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadceb8b-e63c-49f8-b833-27b73ede30f1 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:20.324258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-de18b71c-cd53-4ad1-aa58-62e263d9bf75 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:20.325597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-de18b71c-cd53-4ad1-aa58-62e263d9bf75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:20.372722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:20.384528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f09caa-8bb3-4de3-b4c7-df1cac208856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:20.385140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f09caa-8bb3-4de3-b4c7-df1cac208856 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:20.385416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f09caa-8bb3-4de3-b4c7-df1cac208856 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:20.386234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f09caa-8bb3-4de3-b4c7-df1cac208856 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:20.472234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f09caa-8bb3-4de3-b4c7-df1cac208856 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:20.472895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 381/790] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:20 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:20.473820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f09caa-8bb3-4de3-b4c7-df1cac208856 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:20.567449 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:20.580971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 410/791] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:20 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:27:20.586851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de18b71c-cd53-4ad1-aa58-62e263d9bf75 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:20.741965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de18b71c-cd53-4ad1-aa58-62e263d9bf75 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:20.831034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6dadf1-f8c3-4ef9-857f-07f4cbbf0b50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:20.831761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6dadf1-f8c3-4ef9-857f-07f4cbbf0b50 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:20.832042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6dadf1-f8c3-4ef9-857f-07f4cbbf0b50 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:20.832311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6dadf1-f8c3-4ef9-857f-07f4cbbf0b50 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:20.845909 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6dadf1-f8c3-4ef9-857f-07f4cbbf0b50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:20.848124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6dadf1-f8c3-4ef9-857f-07f4cbbf0b50 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:20.866916 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f09caa-8bb3-4de3-b4c7-df1cac208856 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-de6dadf1-f8c3-4ef9-857f-07f4cbbf0b50 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:20.887973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:20.888975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:20.889855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:20.889855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:20.889855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:20.901920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 382/792] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:20.905042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6dadf1-f8c3-4ef9-857f-07f4cbbf0b50 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:20.941165 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:21.055844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f09caa-8bb3-4de3-b4c7-df1cac208856 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:21.094392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6dadf1-f8c3-4ef9-857f-07f4cbbf0b50 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:21.112793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a70c0e-55bb-484d-984b-4f8a64cc3249 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:21.113795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a70c0e-55bb-484d-984b-4f8a64cc3249 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:21.114310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a70c0e-55bb-484d-984b-4f8a64cc3249 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:21.114770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a70c0e-55bb-484d-984b-4f8a64cc3249 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:21.133456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a70c0e-55bb-484d-984b-4f8a64cc3249 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-e5a70c0e-55bb-484d-984b-4f8a64cc3249 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:21.145086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:21.146012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:21.146929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:21.146929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:21.146929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:21.146929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:21.146929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:21.146929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:21.146929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:21.146929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:21.146929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:21.146929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:21.146929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:21.146929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:21.147571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 411/793] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:21.149146 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f26faac-13bc-437e-b693-7261d4ba63c6 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:21.149579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f26faac-13bc-437e-b693-7261d4ba63c6 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:21.150010 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f26faac-13bc-437e-b693-7261d4ba63c6 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:21.151641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 412/794] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:21.152606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2520c4-c7cb-41a9-8205-8146d6831401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:21.152812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34cf518-9b9d-49d1-ac1b-3c99bf7f23dc None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:21.153046 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34cf518-9b9d-49d1-ac1b-3c99bf7f23dc None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:21.153325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34cf518-9b9d-49d1-ac1b-3c99bf7f23dc None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:21.154126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2520c4-c7cb-41a9-8205-8146d6831401 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:21.158687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 413/795] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:21.158687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34cf518-9b9d-49d1-ac1b-3c99bf7f23dc None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:21.159296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2520c4-c7cb-41a9-8205-8146d6831401 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:21.159978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2520c4-c7cb-41a9-8205-8146d6831401 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:21.160880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2520c4-c7cb-41a9-8205-8146d6831401 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:21.161433 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2520c4-c7cb-41a9-8205-8146d6831401 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:21.163082 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2520c4-c7cb-41a9-8205-8146d6831401 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:21.177886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 383/796] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:21.179240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9386581b-f495-4ce6-a0ab-c8255aac2087 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:21.179578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9386581b-f495-4ce6-a0ab-c8255aac2087 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:21.179905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9386581b-f495-4ce6-a0ab-c8255aac2087 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:21.200116 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-9386581b-f495-4ce6-a0ab-c8255aac2087 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:21.201416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-9386581b-f495-4ce6-a0ab-c8255aac2087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:21.228857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:21.288884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 384/797] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:21 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:21.288884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c256ecd7-308e-4971-8bbe-be41fb3a9311 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:21.288884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c256ecd7-308e-4971-8bbe-be41fb3a9311 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:21.288884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c256ecd7-308e-4971-8bbe-be41fb3a9311 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:21.290719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:21.302617 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-c256ecd7-308e-4971-8bbe-be41fb3a9311 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:21.303899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c256ecd7-308e-4971-8bbe-be41fb3a9311 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:21.308113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34cf518-9b9d-49d1-ac1b-3c99bf7f23dc None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:21.348249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4aa468e-0b0b-4c61-8f13-dc8d1f9e19da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:21.348783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4aa468e-0b0b-4c61-8f13-dc8d1f9e19da None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:21.349059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4aa468e-0b0b-4c61-8f13-dc8d1f9e19da None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:21.349329 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4aa468e-0b0b-4c61-8f13-dc8d1f9e19da None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:21.350018 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4aa468e-0b0b-4c61-8f13-dc8d1f9e19da None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:21.350346 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4aa468e-0b0b-4c61-8f13-dc8d1f9e19da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:21.351771 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4aa468e-0b0b-4c61-8f13-dc8d1f9e19da None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:21.362424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 414/798] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:27:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:21.363747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4aa468e-0b0b-4c61-8f13-dc8d1f9e19da None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:21.490681 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 385/799] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:21 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:21.491742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c256ecd7-308e-4971-8bbe-be41fb3a9311 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:21.492080 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4aa468e-0b0b-4c61-8f13-dc8d1f9e19da None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:21.506530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:21.542070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f03f2b9-5ec9-4db3-9c4e-1e3b0b0dd785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:21.543112 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f03f2b9-5ec9-4db3-9c4e-1e3b0b0dd785 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:21.544049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f03f2b9-5ec9-4db3-9c4e-1e3b0b0dd785 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:21.544527 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f03f2b9-5ec9-4db3-9c4e-1e3b0b0dd785 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:21.552172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f03f2b9-5ec9-4db3-9c4e-1e3b0b0dd785 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:21.553925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f03f2b9-5ec9-4db3-9c4e-1e3b0b0dd785 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-2f03f2b9-5ec9-4db3-9c4e-1e3b0b0dd785 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:21.567113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:21.568027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:21.568919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:21.568919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:21.568919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:21.568919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:21.568919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:21.568919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:21.568919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:21.568919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:21.571496 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 415/800] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:21.577474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f03f2b9-5ec9-4db3-9c4e-1e3b0b0dd785 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:21.657152 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c256ecd7-308e-4971-8bbe-be41fb3a9311 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:21.693749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dee309-3604-4ba1-8d51-71309710ecbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:21.694800 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dee309-3604-4ba1-8d51-71309710ecbf None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:21.695285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dee309-3604-4ba1-8d51-71309710ecbf None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:21.695754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dee309-3604-4ba1-8d51-71309710ecbf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:21.698226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:21.707496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dee309-3604-4ba1-8d51-71309710ecbf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:21.709305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dee309-3604-4ba1-8d51-71309710ecbf None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:21.710428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f03f2b9-5ec9-4db3-9c4e-1e3b0b0dd785 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-c6dee309-3604-4ba1-8d51-71309710ecbf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:21.726069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:21.727177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:21.728083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:21.728083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:21.728083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:21.730141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 386/801] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:21.731241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dee309-3604-4ba1-8d51-71309710ecbf None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:21.756673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc8fdd8-a20e-48bb-984c-6627bfa3bc3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:21.757337 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc8fdd8-a20e-48bb-984c-6627bfa3bc3c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:21.757616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc8fdd8-a20e-48bb-984c-6627bfa3bc3c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:21.757918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc8fdd8-a20e-48bb-984c-6627bfa3bc3c None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:21.758775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc8fdd8-a20e-48bb-984c-6627bfa3bc3c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:21.759175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc8fdd8-a20e-48bb-984c-6627bfa3bc3c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:21.760941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc8fdd8-a20e-48bb-984c-6627bfa3bc3c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:21.783011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 416/802] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:21.783011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc8fdd8-a20e-48bb-984c-6627bfa3bc3c None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:21.817652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:21.851544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dee309-3604-4ba1-8d51-71309710ecbf None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:21.888447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71198c22-2b44-40c9-8b3d-3f9a8f57ef84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:21.888844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc8fdd8-a20e-48bb-984c-6627bfa3bc3c None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:21.889304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71198c22-2b44-40c9-8b3d-3f9a8f57ef84 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:21.889304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71198c22-2b44-40c9-8b3d-3f9a8f57ef84 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:21.889581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71198c22-2b44-40c9-8b3d-3f9a8f57ef84 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:21.901318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71198c22-2b44-40c9-8b3d-3f9a8f57ef84 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-71198c22-2b44-40c9-8b3d-3f9a8f57ef84 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:21.905924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:21.907052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 387/803] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:27:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:21.907957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71198c22-2b44-40c9-8b3d-3f9a8f57ef84 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:21.930343 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cc772c-20a0-401e-b472-3e09645973d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:21.930951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cc772c-20a0-401e-b472-3e09645973d9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:21.931227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cc772c-20a0-401e-b472-3e09645973d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:21.931497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cc772c-20a0-401e-b472-3e09645973d9 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:21.932134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cc772c-20a0-401e-b472-3e09645973d9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:21.932437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cc772c-20a0-401e-b472-3e09645973d9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:21.933804 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cc772c-20a0-401e-b472-3e09645973d9 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:21.945612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 417/804] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:21.946441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cc772c-20a0-401e-b472-3e09645973d9 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:21.964291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:22.021480 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71198c22-2b44-40c9-8b3d-3f9a8f57ef84 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:22.042996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:22.054036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cc772c-20a0-401e-b472-3e09645973d9 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:22.055858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6522d5-0b7c-4f05-9e02-82c85cd18729 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:22.056951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6522d5-0b7c-4f05-9e02-82c85cd18729 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:22.056951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6522d5-0b7c-4f05-9e02-82c85cd18729 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:22.057212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6522d5-0b7c-4f05-9e02-82c85cd18729 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:22.058029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6522d5-0b7c-4f05-9e02-82c85cd18729 None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=e5896018712d4b82a781365e478ee855) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:22.058412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6522d5-0b7c-4f05-9e02-82c85cd18729 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:22.059835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6522d5-0b7c-4f05-9e02-82c85cd18729 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:22.078240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 388/805] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:21 2022] GET /identity/v3/role_assignments?user.id=e5896018712d4b82a781365e478ee855&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=43ca25467bc6484da3a50508efb79230 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:22.078240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6522d5-0b7c-4f05-9e02-82c85cd18729 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:22.105439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9e5544-59cb-4069-8939-8de4580e7df5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:22.106095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9e5544-59cb-4069-8939-8de4580e7df5 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:22.106410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9e5544-59cb-4069-8939-8de4580e7df5 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:22.106685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9e5544-59cb-4069-8939-8de4580e7df5 None admin] PATH_INFO: `/v3/services/network` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:22.107317 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9e5544-59cb-4069-8939-8de4580e7df5 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:22.108933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9e5544-59cb-4069-8939-8de4580e7df5 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-ae9e5544-59cb-4069-8939-8de4580e7df5 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:22.119045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:22.120113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:22.121429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 418/806] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:21 2022] GET /identity/v3/services/network => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:22.122286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9e5544-59cb-4069-8939-8de4580e7df5 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:22.193315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6522d5-0b7c-4f05-9e02-82c85cd18729 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:22.204630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:22.230735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25052d0-5691-4ca5-99c6-e4cb143608f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:22.231413 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25052d0-5691-4ca5-99c6-e4cb143608f3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:22.231689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25052d0-5691-4ca5-99c6-e4cb143608f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:22.231954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25052d0-5691-4ca5-99c6-e4cb143608f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:22.232693 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25052d0-5691-4ca5-99c6-e4cb143608f3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:22.232991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25052d0-5691-4ca5-99c6-e4cb143608f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:22.234525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25052d0-5691-4ca5-99c6-e4cb143608f3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:22.244901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9e5544-59cb-4069-8939-8de4580e7df5 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:22.251008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 389/807] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:27:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:22.251209 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25052d0-5691-4ca5-99c6-e4cb143608f3 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:22.289268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58312aa2-9539-4e36-8540-6f372594ba38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:22.291857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58312aa2-9539-4e36-8540-6f372594ba38 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:22.291857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58312aa2-9539-4e36-8540-6f372594ba38 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:22.291857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58312aa2-9539-4e36-8540-6f372594ba38 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:22.299233 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58312aa2-9539-4e36-8540-6f372594ba38 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:22.301152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58312aa2-9539-4e36-8540-6f372594ba38 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-58312aa2-9539-4e36-8540-6f372594ba38 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:22.312942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:22.313921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:22.315464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:22.315464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:22.315464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:22.315464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:22.315464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:22.315464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:22.315464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:22.315464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:22.315464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 24 03:27:22.315464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:22.315464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 419/808] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:22 2022] GET /identity/v3/users/swift => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:22.316327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58312aa2-9539-4e36-8540-6f372594ba38 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:22.333592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:22.374593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25052d0-5691-4ca5-99c6-e4cb143608f3 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:22.414288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d3d4d4-bb2e-4ba0-a160-12c04342e7bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:22.414902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d3d4d4-bb2e-4ba0-a160-12c04342e7bb None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:22.415175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d3d4d4-bb2e-4ba0-a160-12c04342e7bb None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:22.415448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d3d4d4-bb2e-4ba0-a160-12c04342e7bb None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:22.416085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d3d4d4-bb2e-4ba0-a160-12c04342e7bb None admin] RBAC: Adding query filter params (name=network) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:22.416386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d3d4d4-bb2e-4ba0-a160-12c04342e7bb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:22.417805 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d3d4d4-bb2e-4ba0-a160-12c04342e7bb None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:22.422415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 390/809] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:22 2022] GET /identity/v3/services?name=network => generated 124 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:22.423700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08adf821-30d3-472b-81fb-a100dc7cd897 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:22.423981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08adf821-30d3-472b-81fb-a100dc7cd897 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:22.424254 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08adf821-30d3-472b-81fb-a100dc7cd897 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:22.429921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 391/810] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:22.429921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08adf821-30d3-472b-81fb-a100dc7cd897 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:22.440757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58312aa2-9539-4e36-8540-6f372594ba38 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:22.469700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:22.477962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7847e8ec-ba31-406d-aa19-c8b3e99728c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:22.480724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7847e8ec-ba31-406d-aa19-c8b3e99728c0 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:22.480724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7847e8ec-ba31-406d-aa19-c8b3e99728c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:22.480724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7847e8ec-ba31-406d-aa19-c8b3e99728c0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:22.490914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7847e8ec-ba31-406d-aa19-c8b3e99728c0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:22.492357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7847e8ec-ba31-406d-aa19-c8b3e99728c0 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-7847e8ec-ba31-406d-aa19-c8b3e99728c0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:22.508128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:22.509279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:22.510418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:22.510418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:22.510418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:22.514056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 420/811] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:22.514939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7847e8ec-ba31-406d-aa19-c8b3e99728c0 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:22.541615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08adf821-30d3-472b-81fb-a100dc7cd897 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:22.576744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581cf998-e452-4f30-85df-8dfd15d1fa4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:22.577343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581cf998-e452-4f30-85df-8dfd15d1fa4b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:22.577613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581cf998-e452-4f30-85df-8dfd15d1fa4b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:22.577881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581cf998-e452-4f30-85df-8dfd15d1fa4b None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:22.578635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581cf998-e452-4f30-85df-8dfd15d1fa4b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:22.578937 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581cf998-e452-4f30-85df-8dfd15d1fa4b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:22.580341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581cf998-e452-4f30-85df-8dfd15d1fa4b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:22.585415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:22.599483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 392/812] 10.222.0.65 () {56 vars in 1135 bytes} [Wed Aug 24 03:27:22 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:22.600741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e5138b-d477-4b89-b43a-63820af628af None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:22.601018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e5138b-d477-4b89-b43a-63820af628af None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:22.601282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e5138b-d477-4b89-b43a-63820af628af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:22.618981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-c7e5138b-d477-4b89-b43a-63820af628af None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:22.620172 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c7e5138b-d477-4b89-b43a-63820af628af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:22.630036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7847e8ec-ba31-406d-aa19-c8b3e99728c0 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:22.666669 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8b5463-ff1d-4029-b358-9236c59583cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:22.667290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8b5463-ff1d-4029-b358-9236c59583cd None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:22.667547 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8b5463-ff1d-4029-b358-9236c59583cd None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:22.667814 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8b5463-ff1d-4029-b358-9236c59583cd None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:22.668434 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8b5463-ff1d-4029-b358-9236c59583cd None admin] RBAC: Adding query filter params (type=network) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:22.668729 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8b5463-ff1d-4029-b358-9236c59583cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:22.670087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8b5463-ff1d-4029-b358-9236c59583cd None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:22.675837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 421/813] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:22 2022] GET /identity/v3/services?type=network => generated 350 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:22.681214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8b5463-ff1d-4029-b358-9236c59583cd None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:22.710293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 393/814] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:22 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:22.711793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e5138b-d477-4b89-b43a-63820af628af None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:22.738666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:22.797461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8b5463-ff1d-4029-b358-9236c59583cd None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:22.820924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e5138b-d477-4b89-b43a-63820af628af None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:22.828168 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:22.834571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808340ad-e632-4431-834a-41072465697c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:22.835358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808340ad-e632-4431-834a-41072465697c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:22.841856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808340ad-e632-4431-834a-41072465697c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:22.841856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808340ad-e632-4431-834a-41072465697c None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:22.841856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808340ad-e632-4431-834a-41072465697c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:22.841856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808340ad-e632-4431-834a-41072465697c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:22.841856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808340ad-e632-4431-834a-41072465697c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:22.855651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 422/815] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:22.856894 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808340ad-e632-4431-834a-41072465697c None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:22.860564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f83cdb-93ea-431b-9aaa-4e802edba67e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:22.861188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f83cdb-93ea-431b-9aaa-4e802edba67e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:22.861504 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f83cdb-93ea-431b-9aaa-4e802edba67e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:22.862352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f83cdb-93ea-431b-9aaa-4e802edba67e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:22.881631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f83cdb-93ea-431b-9aaa-4e802edba67e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:22.883181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f83cdb-93ea-431b-9aaa-4e802edba67e None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-80f83cdb-93ea-431b-9aaa-4e802edba67e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:22.899356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:22.900309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:22.901213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:22.901213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:22.901213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:22.905362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 394/816] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:22.906370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f83cdb-93ea-431b-9aaa-4e802edba67e None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:22.965040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808340ad-e632-4431-834a-41072465697c None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:22.994450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:23.000072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b0c1aa-388f-4bc8-88aa-9b06cf142987 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:23.000383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b0c1aa-388f-4bc8-88aa-9b06cf142987 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:23.000660 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b0c1aa-388f-4bc8-88aa-9b06cf142987 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:23.000932 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b0c1aa-388f-4bc8-88aa-9b06cf142987 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:23.001574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b0c1aa-388f-4bc8-88aa-9b06cf142987 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:23.003637 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b0c1aa-388f-4bc8-88aa-9b06cf142987 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:23.020328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f83cdb-93ea-431b-9aaa-4e802edba67e None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:23.054589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed6d955-26b0-486c-b730-9e62521ed034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:23.055207 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed6d955-26b0-486c-b730-9e62521ed034 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:23.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed6d955-26b0-486c-b730-9e62521ed034 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:23.055741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed6d955-26b0-486c-b730-9e62521ed034 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:23.063110 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed6d955-26b0-486c-b730-9e62521ed034 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:23.064522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed6d955-26b0-486c-b730-9e62521ed034 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-1ed6d955-26b0-486c-b730-9e62521ed034 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:23.075987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:23.076947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:23.077907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:23.077907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:23.077907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:23.077907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:23.077907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:23.077907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:23.077907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:23.077907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:23.077907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 03:27:23.077907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:23.077907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 395/817] 10.222.0.65 () {56 vars in 1117 bytes} [Wed Aug 24 03:27:22 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:23.077907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed6d955-26b0-486c-b730-9e62521ed034 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:23.100330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 423/818] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:27:22 2022] POST /identity/v3/endpoints => generated 337 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:23.105534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b0c1aa-388f-4bc8-88aa-9b06cf142987 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:23.169928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:23.234985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed6d955-26b0-486c-b730-9e62521ed034 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:23.251169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b0c1aa-388f-4bc8-88aa-9b06cf142987 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:23.285472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea5de3c-b207-444b-91b3-f4866537b9b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:23.286738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5de3c-b207-444b-91b3-f4866537b9b3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:23.287203 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5de3c-b207-444b-91b3-f4866537b9b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:23.287625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5de3c-b207-444b-91b3-f4866537b9b3 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:23.288544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea5de3c-b207-444b-91b3-f4866537b9b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:23.289000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea5de3c-b207-444b-91b3-f4866537b9b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:23.290979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea5de3c-b207-444b-91b3-f4866537b9b3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:23.291461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26474721-9690-4a53-9a4e-1ec6460e229d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:23.292116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26474721-9690-4a53-9a4e-1ec6460e229d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:23.292398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26474721-9690-4a53-9a4e-1ec6460e229d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:23.292674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26474721-9690-4a53-9a4e-1ec6460e229d None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:23.293324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26474721-9690-4a53-9a4e-1ec6460e229d None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:23.294788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26474721-9690-4a53-9a4e-1ec6460e229d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:23.295179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-26474721-9690-4a53-9a4e-1ec6460e229d None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:23.305506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:23.306876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:23.307715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 424/819] 10.222.0.65 () {56 vars in 1129 bytes} [Wed Aug 24 03:27:23 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:23.308501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26474721-9690-4a53-9a4e-1ec6460e229d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:23.311620 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 396/820] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:23.317732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea5de3c-b207-444b-91b3-f4866537b9b3 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:23.438898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26474721-9690-4a53-9a4e-1ec6460e229d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:23.442421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea5de3c-b207-444b-91b3-f4866537b9b3 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:23.446594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:23.477658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceeaddbb-7e36-4458-b913-84934047620d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:23.478291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeaddbb-7e36-4458-b913-84934047620d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:23.478596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeaddbb-7e36-4458-b913-84934047620d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:23.478865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeaddbb-7e36-4458-b913-84934047620d None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:23.479165 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f390125-751c-4670-a408-24fc6680261b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:23.479486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceeaddbb-7e36-4458-b913-84934047620d None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:23.479762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f390125-751c-4670-a408-24fc6680261b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:23.479859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceeaddbb-7e36-4458-b913-84934047620d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:23.480038 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f390125-751c-4670-a408-24fc6680261b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:23.480302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f390125-751c-4670-a408-24fc6680261b None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:23.481282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f390125-751c-4670-a408-24fc6680261b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:23.481396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceeaddbb-7e36-4458-b913-84934047620d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:23.481635 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f390125-751c-4670-a408-24fc6680261b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:23.483546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f390125-751c-4670-a408-24fc6680261b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:23.486053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 397/821] 10.222.0.65 () {56 vars in 1124 bytes} [Wed Aug 24 03:27:23 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:23.486664 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceeaddbb-7e36-4458-b913-84934047620d None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:23.511052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 425/822] 10.222.0.65 () {56 vars in 1149 bytes} [Wed Aug 24 03:27:23 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:23.511052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f390125-751c-4670-a408-24fc6680261b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:23.553125 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:23.599249 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceeaddbb-7e36-4458-b913-84934047620d None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:23.619297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f390125-751c-4670-a408-24fc6680261b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:23.635446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec64e5cb-dea5-432d-82d1-bab68ed54aa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:23.636078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec64e5cb-dea5-432d-82d1-bab68ed54aa3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:23.636358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec64e5cb-dea5-432d-82d1-bab68ed54aa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:23.636629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec64e5cb-dea5-432d-82d1-bab68ed54aa3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:23.643411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec64e5cb-dea5-432d-82d1-bab68ed54aa3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:23.644949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec64e5cb-dea5-432d-82d1-bab68ed54aa3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:23.654704 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9925291f-f576-482d-a247-b4f49fbea1f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:23.655306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9925291f-f576-482d-a247-b4f49fbea1f2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:23.655580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9925291f-f576-482d-a247-b4f49fbea1f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:23.655847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9925291f-f576-482d-a247-b4f49fbea1f2 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-ec64e5cb-dea5-432d-82d1-bab68ed54aa3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:23.656177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:23.657051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:23.663830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:23.663830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:23.663830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:23.663830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:23.663830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:23.663830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:23.663830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:23.663830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:23.664306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9925291f-f576-482d-a247-b4f49fbea1f2 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:23.664306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9925291f-f576-482d-a247-b4f49fbea1f2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:23.664436 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 398/823] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:23.664504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9925291f-f576-482d-a247-b4f49fbea1f2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:23.664571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec64e5cb-dea5-432d-82d1-bab68ed54aa3 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:23.669845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 426/824] 10.222.0.65 () {56 vars in 1124 bytes} [Wed Aug 24 03:27:23 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:23.675568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9925291f-f576-482d-a247-b4f49fbea1f2 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:23.704121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:23.767717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec64e5cb-dea5-432d-82d1-bab68ed54aa3 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:23.782263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9925291f-f576-482d-a247-b4f49fbea1f2 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:23.790019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:23.805745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b4fe9-3ef2-49bf-a3a5-52b2d9796c8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:23.806431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b4fe9-3ef2-49bf-a3a5-52b2d9796c8c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:23.806709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b4fe9-3ef2-49bf-a3a5-52b2d9796c8c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:23.806983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b4fe9-3ef2-49bf-a3a5-52b2d9796c8c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:23.819386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9b4fe9-3ef2-49bf-a3a5-52b2d9796c8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:23.821256 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9b4fe9-3ef2-49bf-a3a5-52b2d9796c8c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:23.821736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b07e7e-8df7-4a01-9c8d-2ba8da458a7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:23.822365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b07e7e-8df7-4a01-9c8d-2ba8da458a7f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:23.822641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b07e7e-8df7-4a01-9c8d-2ba8da458a7f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:23.822906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b07e7e-8df7-4a01-9c8d-2ba8da458a7f None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:23.823580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b07e7e-8df7-4a01-9c8d-2ba8da458a7f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:23.823883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b07e7e-8df7-4a01-9c8d-2ba8da458a7f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:23.825282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b07e7e-8df7-4a01-9c8d-2ba8da458a7f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-af9b4fe9-3ef2-49bf-a3a5-52b2d9796c8c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:23.836739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:23.837719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:23.848771 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:23.848771 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:23.848771 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:23.848984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 427/826] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:23.849071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 399/826] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:23.849152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b07e7e-8df7-4a01-9c8d-2ba8da458a7f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:23.851258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b4fe9-3ef2-49bf-a3a5-52b2d9796c8c None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:23.963863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b4fe9-3ef2-49bf-a3a5-52b2d9796c8c None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:23.999228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a76634-09cc-4e2f-b849-f98b0ac03762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:23.999830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a76634-09cc-4e2f-b849-f98b0ac03762 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:24.000100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a76634-09cc-4e2f-b849-f98b0ac03762 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:24.000367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a76634-09cc-4e2f-b849-f98b0ac03762 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:24.000999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a76634-09cc-4e2f-b849-f98b0ac03762 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:24.001291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a76634-09cc-4e2f-b849-f98b0ac03762 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:24.003452 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a76634-09cc-4e2f-b849-f98b0ac03762 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:24.016675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 400/827] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:24.017284 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a76634-09cc-4e2f-b849-f98b0ac03762 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:24.056785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:24.062008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:24.075540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b07e7e-8df7-4a01-9c8d-2ba8da458a7f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:24.109448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2199653c-070d-4988-96da-0dc9c7b4569d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:24.110050 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2199653c-070d-4988-96da-0dc9c7b4569d None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:24.110346 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2199653c-070d-4988-96da-0dc9c7b4569d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:24.110612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2199653c-070d-4988-96da-0dc9c7b4569d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:24.111256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2199653c-070d-4988-96da-0dc9c7b4569d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:24.113555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2199653c-070d-4988-96da-0dc9c7b4569d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:24.128110 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a76634-09cc-4e2f-b849-f98b0ac03762 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:24.161633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c93a7c3-aafe-4d4c-a563-0dfb14ed29ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:24.162321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c93a7c3-aafe-4d4c-a563-0dfb14ed29ff None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:24.162600 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c93a7c3-aafe-4d4c-a563-0dfb14ed29ff None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:24.162871 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c93a7c3-aafe-4d4c-a563-0dfb14ed29ff None admin] PATH_INFO: `/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/4591bec74a4641af818b2d75b657115e/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:24.164801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c93a7c3-aafe-4d4c-a563-0dfb14ed29ff None admin] RBAC: Authorizing `identity:create_grant(project_id=5c653d5fa4304f298a295ce635efd5e6, user_id=4591bec74a4641af818b2d75b657115e, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:24.166238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c93a7c3-aafe-4d4c-a563-0dfb14ed29ff None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:24.172828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 428/828] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:27:23 2022] POST /identity/v3/endpoints => generated 346 bytes in 334 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:24.173640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2199653c-070d-4988-96da-0dc9c7b4569d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:24.242985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 401/829] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:27:24 2022] PUT /identity/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/4591bec74a4641af818b2d75b657115e/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:27:24.251699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c568d5f-db4d-448a-bd8e-8cbb01051cdb None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:24.252034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c568d5f-db4d-448a-bd8e-8cbb01051cdb None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:24.252251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c568d5f-db4d-448a-bd8e-8cbb01051cdb None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:24.270207 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 402/830] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:24 2022] GET /identity => generated 270 bytes in 23 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:24.273021 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e239936-498f-4101-99ae-67b06fd07428 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:24.273306 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e239936-498f-4101-99ae-67b06fd07428 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:24.273581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e239936-498f-4101-99ae-67b06fd07428 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:24.298357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2199653c-070d-4988-96da-0dc9c7b4569d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:24.322203 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1e239936-498f-4101-99ae-67b06fd07428 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:24.322903 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1e239936-498f-4101-99ae-67b06fd07428 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:24.342514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e82133-ef28-49b5-b094-be88cd08c25e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:24.345212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e82133-ef28-49b5-b094-be88cd08c25e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:24.345212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e82133-ef28-49b5-b094-be88cd08c25e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:24.345212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e82133-ef28-49b5-b094-be88cd08c25e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:24.352843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e82133-ef28-49b5-b094-be88cd08c25e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:24.354408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e82133-ef28-49b5-b094-be88cd08c25e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:24.357462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-19e82133-ef28-49b5-b094-be88cd08c25e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:24.370003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:24.374863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:24.375770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:24.375770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:24.375770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:24.375770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:24.375770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:24.375770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:24.375770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:24.375770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:24.375770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 429/831] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:24.378672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e82133-ef28-49b5-b094-be88cd08c25e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:24.430226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 403/832] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:24 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:24.506060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e82133-ef28-49b5-b094-be88cd08c25e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:24.540967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5704aaa-dae6-4a83-baf3-8c00de0aa722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:24.541662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5704aaa-dae6-4a83-baf3-8c00de0aa722 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:24.542012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5704aaa-dae6-4a83-baf3-8c00de0aa722 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:24.542380 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5704aaa-dae6-4a83-baf3-8c00de0aa722 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:24.543150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5704aaa-dae6-4a83-baf3-8c00de0aa722 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:24.543520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5704aaa-dae6-4a83-baf3-8c00de0aa722 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:24.545178 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5704aaa-dae6-4a83-baf3-8c00de0aa722 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:24.558516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 430/833] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:24.564612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e239936-498f-4101-99ae-67b06fd07428 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:24.589899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:24.670033 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e239936-498f-4101-99ae-67b06fd07428 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:24.706501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5704aaa-dae6-4a83-baf3-8c00de0aa722 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:24.707609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cf0722-7a41-49e6-96f3-e1c9428882af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:24.708330 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf0722-7a41-49e6-96f3-e1c9428882af None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:24.708610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf0722-7a41-49e6-96f3-e1c9428882af None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:24.708934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf0722-7a41-49e6-96f3-e1c9428882af None admin] PATH_INFO: `/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/c63ade06fed946e79dc1e4b5ecfab5e6/roles/94618a16f2e6449ba1cd5fbbd7c19f56` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:24.724815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cf0722-7a41-49e6-96f3-e1c9428882af None admin] RBAC: Authorizing `identity:create_grant(project_id=5c653d5fa4304f298a295ce635efd5e6, user_id=c63ade06fed946e79dc1e4b5ecfab5e6, role_id=94618a16f2e6449ba1cd5fbbd7c19f56)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:24.726299 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cf0722-7a41-49e6-96f3-e1c9428882af None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:24.772880 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:24.813196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5704aaa-dae6-4a83-baf3-8c00de0aa722 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:24.831054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 404/834] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:27:24 2022] PUT /identity/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/c63ade06fed946e79dc1e4b5ecfab5e6/roles/94618a16f2e6449ba1cd5fbbd7c19f56 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:27:24.858976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5790745-a406-4ae4-9f2f-e3e1c34447f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:24.859679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5790745-a406-4ae4-9f2f-e3e1c34447f2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:24.859964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5790745-a406-4ae4-9f2f-e3e1c34447f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:24.860325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5790745-a406-4ae4-9f2f-e3e1c34447f2 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:24.860957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5790745-a406-4ae4-9f2f-e3e1c34447f2 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:24.865942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5790745-a406-4ae4-9f2f-e3e1c34447f2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-d5790745-a406-4ae4-9f2f-e3e1c34447f2 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:24.873252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:24.874458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:24.875351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 431/835] 10.222.0.65 () {56 vars in 1132 bytes} [Wed Aug 24 03:27:24 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:24.883405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cf0722-7a41-49e6-96f3-e1c9428882af None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:24.971440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:25.007970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cf0722-7a41-49e6-96f3-e1c9428882af None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:25.045436 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795f0075-3aee-4875-b101-0143b20e3730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:25.046506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795f0075-3aee-4875-b101-0143b20e3730 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:25.047012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795f0075-3aee-4875-b101-0143b20e3730 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:25.047716 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795f0075-3aee-4875-b101-0143b20e3730 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:25.048544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795f0075-3aee-4875-b101-0143b20e3730 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:25.049044 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795f0075-3aee-4875-b101-0143b20e3730 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:25.050819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795f0075-3aee-4875-b101-0143b20e3730 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:25.056062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 405/836] 10.222.0.65 () {56 vars in 1126 bytes} [Wed Aug 24 03:27:24 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:25.061881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5790745-a406-4ae4-9f2f-e3e1c34447f2 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:25.168556 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5790745-a406-4ae4-9f2f-e3e1c34447f2 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:25.193516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:25.204599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604adf33-46c2-4361-8e86-6f052a21599c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:25.205216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604adf33-46c2-4361-8e86-6f052a21599c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:25.205563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604adf33-46c2-4361-8e86-6f052a21599c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:25.205877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604adf33-46c2-4361-8e86-6f052a21599c None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:25.206545 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604adf33-46c2-4361-8e86-6f052a21599c None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:25.206845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604adf33-46c2-4361-8e86-6f052a21599c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:25.208206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604adf33-46c2-4361-8e86-6f052a21599c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:25.213150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 432/837] 10.222.0.65 () {56 vars in 1126 bytes} [Wed Aug 24 03:27:25 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:26.229578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57c3c62-bc56-4e8d-8619-9089a3edd582 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:26.230103 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57c3c62-bc56-4e8d-8619-9089a3edd582 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:26.230418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57c3c62-bc56-4e8d-8619-9089a3edd582 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:26.232138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 406/838] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:26.238256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b623eec-450c-47f0-bd37-a9345c11c42f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:26.238700 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b623eec-450c-47f0-bd37-a9345c11c42f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:26.239024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b623eec-450c-47f0-bd37-a9345c11c42f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:26.257926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5b623eec-450c-47f0-bd37-a9345c11c42f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:26.259161 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5b623eec-450c-47f0-bd37-a9345c11c42f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:26.272802 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bbd76-8aa3-4bb1-ac91-44a905bd74c2 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:26.273103 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bbd76-8aa3-4bb1-ac91-44a905bd74c2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:26.273350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bbd76-8aa3-4bb1-ac91-44a905bd74c2 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:26.275154 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 407/839] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:26.280786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e3c322-4e74-406f-900d-5722d2077890 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:26.281066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e3c322-4e74-406f-900d-5722d2077890 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:26.281333 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e3c322-4e74-406f-900d-5722d2077890 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:26.301813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-37e3c322-4e74-406f-900d-5722d2077890 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:26.303068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-37e3c322-4e74-406f-900d-5722d2077890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:26.351129 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 433/840] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:26 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:26.381876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 408/841] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:26 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:26.620207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b623eec-450c-47f0-bd37-a9345c11c42f None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:26.658332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e3c322-4e74-406f-900d-5722d2077890 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:26.724003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b623eec-450c-47f0-bd37-a9345c11c42f None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:26.742443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:26.743194 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:26.760872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5720154-9087-4cb6-a08d-92fceaa77104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:26.761497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5720154-9087-4cb6-a08d-92fceaa77104 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:26.775555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5720154-9087-4cb6-a08d-92fceaa77104 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:26.775555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5720154-9087-4cb6-a08d-92fceaa77104 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:26.775555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5720154-9087-4cb6-a08d-92fceaa77104 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:26.775555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5720154-9087-4cb6-a08d-92fceaa77104 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:26.775555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-e5720154-9087-4cb6-a08d-92fceaa77104 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 03:27:26.775555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:26.775555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:26.775555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:26.775555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:26.776151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:26.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:26.777924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:26.777924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:26.777924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:26.777924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:26.777924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:26.777924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:26.777924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 03:27:26.777924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:26.777924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 434/842] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:27:26 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:26.780041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5720154-9087-4cb6-a08d-92fceaa77104 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:26.787721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e3c322-4e74-406f-900d-5722d2077890 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:26.843102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf655c3-4d94-43c2-bde3-218c9c63973e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:26.843731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf655c3-4d94-43c2-bde3-218c9c63973e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:26.844155 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf655c3-4d94-43c2-bde3-218c9c63973e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:26.844425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf655c3-4d94-43c2-bde3-218c9c63973e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:26.858396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf655c3-4d94-43c2-bde3-218c9c63973e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-aaf655c3-4d94-43c2-bde3-218c9c63973e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:26.864076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:26.865014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:26.865870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:26.865870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:26.865870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:26.865870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:26.865870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:26.865870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:26.865870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:26.865870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:26.865870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:26.865870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:26.865870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:26.865870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:26.870481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 409/843] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:26.871290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf655c3-4d94-43c2-bde3-218c9c63973e None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:26.897506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5720154-9087-4cb6-a08d-92fceaa77104 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:26.934137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba87527-f8b7-49ff-98e1-d8dcba1a0f7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:26.935101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba87527-f8b7-49ff-98e1-d8dcba1a0f7e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:26.935534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba87527-f8b7-49ff-98e1-d8dcba1a0f7e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:26.935810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba87527-f8b7-49ff-98e1-d8dcba1a0f7e None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:26.936435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba87527-f8b7-49ff-98e1-d8dcba1a0f7e None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:26.936733 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba87527-f8b7-49ff-98e1-d8dcba1a0f7e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:26.938087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba87527-f8b7-49ff-98e1-d8dcba1a0f7e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:26.943782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 435/844] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:27:26 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:26.944724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f647d7c-df7b-4526-8ede-5d73c33f884c None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:26.945004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f647d7c-df7b-4526-8ede-5d73c33f884c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:26.945275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f647d7c-df7b-4526-8ede-5d73c33f884c None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:26.950366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 436/845] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:26.950366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f647d7c-df7b-4526-8ede-5d73c33f884c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:26.986359 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf655c3-4d94-43c2-bde3-218c9c63973e None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:26.998099 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:27.014810 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:27.024009 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9615a880-e013-4757-8ab4-e73666ec9166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:27.024608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9615a880-e013-4757-8ab4-e73666ec9166 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:27.024881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9615a880-e013-4757-8ab4-e73666ec9166 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:27.025145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9615a880-e013-4757-8ab4-e73666ec9166 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:27.025797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9615a880-e013-4757-8ab4-e73666ec9166 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:27.026094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9615a880-e013-4757-8ab4-e73666ec9166 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:27.027470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9615a880-e013-4757-8ab4-e73666ec9166 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:27.040612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 410/846] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:27:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:27.040612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936b11e4-a1a0-4396-a6bc-6632d3e352a4 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:27.040612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936b11e4-a1a0-4396-a6bc-6632d3e352a4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:27.040612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936b11e4-a1a0-4396-a6bc-6632d3e352a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:27.058075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-936b11e4-a1a0-4396-a6bc-6632d3e352a4 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:27.060260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-936b11e4-a1a0-4396-a6bc-6632d3e352a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:27.062748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f647d7c-df7b-4526-8ede-5d73c33f884c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:27.099725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09417955-a4e2-4f2c-8957-0fb18939bcaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:27.100330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09417955-a4e2-4f2c-8957-0fb18939bcaf None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:27.100606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09417955-a4e2-4f2c-8957-0fb18939bcaf None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:27.100879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09417955-a4e2-4f2c-8957-0fb18939bcaf None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:27.101510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09417955-a4e2-4f2c-8957-0fb18939bcaf None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:27.101806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09417955-a4e2-4f2c-8957-0fb18939bcaf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:27.103198 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09417955-a4e2-4f2c-8957-0fb18939bcaf None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:27.108121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 437/847] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:27:26 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:27.108829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09417955-a4e2-4f2c-8957-0fb18939bcaf None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:27.145762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 411/848] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:27 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:27.147593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17faafa7-07d9-4d14-8998-679197218995 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:27.148106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17faafa7-07d9-4d14-8998-679197218995 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:27.148578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17faafa7-07d9-4d14-8998-679197218995 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:27.150079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 412/849] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:27.155847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a237cfa-6bd6-4f06-b730-1662cde44f68 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:27.156675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a237cfa-6bd6-4f06-b730-1662cde44f68 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:27.157180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a237cfa-6bd6-4f06-b730-1662cde44f68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:27.177182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-5a237cfa-6bd6-4f06-b730-1662cde44f68 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:27.178866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5a237cfa-6bd6-4f06-b730-1662cde44f68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:27.243428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09417955-a4e2-4f2c-8957-0fb18939bcaf None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:27.260753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:27.264837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 413/850] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:27 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:27.285627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b910bcfd-cfbe-4405-aa3b-8572aaed75b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:27.286346 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b910bcfd-cfbe-4405-aa3b-8572aaed75b8 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:27.286631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b910bcfd-cfbe-4405-aa3b-8572aaed75b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:27.286896 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b910bcfd-cfbe-4405-aa3b-8572aaed75b8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:27.298317 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b910bcfd-cfbe-4405-aa3b-8572aaed75b8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:27.307257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b910bcfd-cfbe-4405-aa3b-8572aaed75b8 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-b910bcfd-cfbe-4405-aa3b-8572aaed75b8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:27.317128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:27.318075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:27.319043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:27.319043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:27.319043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:27.322179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 438/851] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:27.324625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a237cfa-6bd6-4f06-b730-1662cde44f68 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:27.378080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:27.430928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a237cfa-6bd6-4f06-b730-1662cde44f68 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:27.432309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b910bcfd-cfbe-4405-aa3b-8572aaed75b8 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:27.475649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225469c8-83c7-4a70-b88d-1a6ec9f57ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:27.476261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225469c8-83c7-4a70-b88d-1a6ec9f57ece None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:27.476537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225469c8-83c7-4a70-b88d-1a6ec9f57ece None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:27.476809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225469c8-83c7-4a70-b88d-1a6ec9f57ece None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:27.477454 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225469c8-83c7-4a70-b88d-1a6ec9f57ece None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:27.478184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225469c8-83c7-4a70-b88d-1a6ec9f57ece None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:27.479928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225469c8-83c7-4a70-b88d-1a6ec9f57ece None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:27.494751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 414/852] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:27.503689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225469c8-83c7-4a70-b88d-1a6ec9f57ece None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:27.543894 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b910bcfd-cfbe-4405-aa3b-8572aaed75b8 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:27.579675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d4c383-6c85-4b81-8513-58b99c2ee100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:27.580309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d4c383-6c85-4b81-8513-58b99c2ee100 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:27.580649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d4c383-6c85-4b81-8513-58b99c2ee100 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:27.580915 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d4c383-6c85-4b81-8513-58b99c2ee100 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:27.591208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d4c383-6c85-4b81-8513-58b99c2ee100 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-f9d4c383-6c85-4b81-8513-58b99c2ee100 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:27.596852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:27.597794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:27.598692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 439/853] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:27:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:27.599396 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d4c383-6c85-4b81-8513-58b99c2ee100 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:27.623531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225469c8-83c7-4a70-b88d-1a6ec9f57ece None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:27.627132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:27.633921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:27.658890 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9067121b-49ab-4bc1-b729-2d4875a9f5fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:27.659513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9067121b-49ab-4bc1-b729-2d4875a9f5fe None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:27.659792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9067121b-49ab-4bc1-b729-2d4875a9f5fe None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:27.660059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9067121b-49ab-4bc1-b729-2d4875a9f5fe None admin] PATH_INFO: `/v3/users/swift` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:27.667636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9067121b-49ab-4bc1-b729-2d4875a9f5fe None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:27.669138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9067121b-49ab-4bc1-b729-2d4875a9f5fe None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-9067121b-49ab-4bc1-b729-2d4875a9f5fe None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:27.680978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:27.682039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:27.683018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:27.683018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:27.683018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:27.683018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:27.683018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:27.683018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:27.683018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:27.683018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:27.683018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 24 03:27:27.683018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:27.686918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 415/854] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:27 2022] GET /identity/v3/users/swift => generated 83 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:27.687630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9067121b-49ab-4bc1-b729-2d4875a9f5fe None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:27.708451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d4c383-6c85-4b81-8513-58b99c2ee100 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:27.743841 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34f152a-53ad-46b3-8eb1-aa5d664ef691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:27.744671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34f152a-53ad-46b3-8eb1-aa5d664ef691 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:27.744783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34f152a-53ad-46b3-8eb1-aa5d664ef691 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:27.744971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34f152a-53ad-46b3-8eb1-aa5d664ef691 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:27.745615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34f152a-53ad-46b3-8eb1-aa5d664ef691 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:27.747010 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34f152a-53ad-46b3-8eb1-aa5d664ef691 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:27.793641 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9067121b-49ab-4bc1-b729-2d4875a9f5fe None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:27.819583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 440/855] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:27:27 2022] POST /identity/v3/services => generated 269 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:27.820660 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34f152a-53ad-46b3-8eb1-aa5d664ef691 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:27.850506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7e089f-4bc3-47b4-9eab-b2262948fee7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:27.852030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7e089f-4bc3-47b4-9eab-b2262948fee7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:27.852030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7e089f-4bc3-47b4-9eab-b2262948fee7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:27.852030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7e089f-4bc3-47b4-9eab-b2262948fee7 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:27.852873 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7e089f-4bc3-47b4-9eab-b2262948fee7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:27.853230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7e089f-4bc3-47b4-9eab-b2262948fee7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:27.854809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7e089f-4bc3-47b4-9eab-b2262948fee7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:27.871447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 416/856] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:27:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:27.872465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7e089f-4bc3-47b4-9eab-b2262948fee7 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:27.907981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:27.920439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:27.940480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34f152a-53ad-46b3-8eb1-aa5d664ef691 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:27.983299 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8128b2-d8ab-48e9-a2f7-96bb41e906c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:27.983906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8128b2-d8ab-48e9-a2f7-96bb41e906c1 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:27.984184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8128b2-d8ab-48e9-a2f7-96bb41e906c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:27.984456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8128b2-d8ab-48e9-a2f7-96bb41e906c1 None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:27.985191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8128b2-d8ab-48e9-a2f7-96bb41e906c1 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:27.985495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8128b2-d8ab-48e9-a2f7-96bb41e906c1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:27.986997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8128b2-d8ab-48e9-a2f7-96bb41e906c1 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:27.996175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7e089f-4bc3-47b4-9eab-b2262948fee7 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:28.010138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 441/857] 10.222.0.65 () {56 vars in 1135 bytes} [Wed Aug 24 03:27:27 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:28.015622 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8128b2-d8ab-48e9-a2f7-96bb41e906c1 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:28.044328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df49fa50-e3f9-4361-bd80-6e9da8d5e26d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:28.044970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df49fa50-e3f9-4361-bd80-6e9da8d5e26d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:28.045348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df49fa50-e3f9-4361-bd80-6e9da8d5e26d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:28.045624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df49fa50-e3f9-4361-bd80-6e9da8d5e26d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:28.056513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df49fa50-e3f9-4361-bd80-6e9da8d5e26d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:28.057963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df49fa50-e3f9-4361-bd80-6e9da8d5e26d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-df49fa50-e3f9-4361-bd80-6e9da8d5e26d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:28.073732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:28.074779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:28.075701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:28.075701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:28.075701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:28.075701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 417/858] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:28.081776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df49fa50-e3f9-4361-bd80-6e9da8d5e26d None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:28.143819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8128b2-d8ab-48e9-a2f7-96bb41e906c1 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:28.176928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:28.189652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e83b09-4700-44fc-b62e-7596a239d473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:28.193159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e83b09-4700-44fc-b62e-7596a239d473 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:28.193159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e83b09-4700-44fc-b62e-7596a239d473 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:28.193159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e83b09-4700-44fc-b62e-7596a239d473 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:28.200002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df49fa50-e3f9-4361-bd80-6e9da8d5e26d None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:28.206572 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:28.238739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462f6bfb-beda-43e2-a49e-dbfa7b884e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:28.239548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462f6bfb-beda-43e2-a49e-dbfa7b884e8f None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:28.239829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462f6bfb-beda-43e2-a49e-dbfa7b884e8f None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:28.240099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462f6bfb-beda-43e2-a49e-dbfa7b884e8f None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:28.240745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462f6bfb-beda-43e2-a49e-dbfa7b884e8f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:28.241047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462f6bfb-beda-43e2-a49e-dbfa7b884e8f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:28.242508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462f6bfb-beda-43e2-a49e-dbfa7b884e8f None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:28.257986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 418/859] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:28.264059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462f6bfb-beda-43e2-a49e-dbfa7b884e8f None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:28.280718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e83b09-4700-44fc-b62e-7596a239d473 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:28.282372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e83b09-4700-44fc-b62e-7596a239d473 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:28.384702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462f6bfb-beda-43e2-a49e-dbfa7b884e8f None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:28.414243 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 442/860] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:28 2022] GET /identity/v3/auth/tokens => generated 1271 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 24 03:27:28.419436 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e83b09-4700-44fc-b62e-7596a239d473 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:28.427728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3211d9be-f666-45aa-a223-49607d4fc177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:28.429101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3211d9be-f666-45aa-a223-49607d4fc177 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:28.429588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3211d9be-f666-45aa-a223-49607d4fc177 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:28.430061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3211d9be-f666-45aa-a223-49607d4fc177 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:28.438937 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3211d9be-f666-45aa-a223-49607d4fc177 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:28.441427 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3211d9be-f666-45aa-a223-49607d4fc177 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:28.443822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-3211d9be-f666-45aa-a223-49607d4fc177 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:28.456146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:28.457117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:28.458075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:28.458075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:28.458075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:28.458075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:28.458075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:28.458075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:28.458075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:28.458075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:28.458075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 03:27:28.458075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:28.465324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:28.466134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 419/861] 10.222.0.65 () {56 vars in 1117 bytes} [Wed Aug 24 03:27:28 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:28.471645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3211d9be-f666-45aa-a223-49607d4fc177 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:28.523424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e83b09-4700-44fc-b62e-7596a239d473 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:28.560186 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e66b4e3-51fb-41f6-b080-42c75c85d2e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:28.560827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e66b4e3-51fb-41f6-b080-42c75c85d2e2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:28.561107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e66b4e3-51fb-41f6-b080-42c75c85d2e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:28.561378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e66b4e3-51fb-41f6-b080-42c75c85d2e2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:28.572606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e66b4e3-51fb-41f6-b080-42c75c85d2e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:28.574063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e66b4e3-51fb-41f6-b080-42c75c85d2e2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:28.581634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3211d9be-f666-45aa-a223-49607d4fc177 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-1e66b4e3-51fb-41f6-b080-42c75c85d2e2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:28.589412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:28.590405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:28.591319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:28.591319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:28.591319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:28.594288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 443/862] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:28.596395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e66b4e3-51fb-41f6-b080-42c75c85d2e2 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:28.621569 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880066c0-af9b-4efc-841d-bc9a2f331e11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:28.622741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880066c0-af9b-4efc-841d-bc9a2f331e11 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:28.623279 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880066c0-af9b-4efc-841d-bc9a2f331e11 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:28.623857 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880066c0-af9b-4efc-841d-bc9a2f331e11 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:28.624881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880066c0-af9b-4efc-841d-bc9a2f331e11 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:28.625411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880066c0-af9b-4efc-841d-bc9a2f331e11 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:28.627329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880066c0-af9b-4efc-841d-bc9a2f331e11 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:28.652739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 420/863] 10.222.0.65 () {56 vars in 1149 bytes} [Wed Aug 24 03:27:28 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:28.654030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880066c0-af9b-4efc-841d-bc9a2f331e11 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:28.699788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:28.709797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e66b4e3-51fb-41f6-b080-42c75c85d2e2 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:28.716671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:28.749106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809a17a1-c0f3-4a77-986f-0237a6ef6de4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:28.750145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809a17a1-c0f3-4a77-986f-0237a6ef6de4 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:28.750670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809a17a1-c0f3-4a77-986f-0237a6ef6de4 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:28.751146 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809a17a1-c0f3-4a77-986f-0237a6ef6de4 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:28.751994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809a17a1-c0f3-4a77-986f-0237a6ef6de4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:28.752643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809a17a1-c0f3-4a77-986f-0237a6ef6de4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:28.754263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809a17a1-c0f3-4a77-986f-0237a6ef6de4 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:28.770259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 444/864] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:28.775997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809a17a1-c0f3-4a77-986f-0237a6ef6de4 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:28.780229 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880066c0-af9b-4efc-841d-bc9a2f331e11 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:28.818035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72016d20-d4eb-43aa-9b2f-8ca33096147a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:28.819090 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72016d20-d4eb-43aa-9b2f-8ca33096147a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:28.819570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72016d20-d4eb-43aa-9b2f-8ca33096147a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:28.820084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72016d20-d4eb-43aa-9b2f-8ca33096147a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:28.897142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72016d20-d4eb-43aa-9b2f-8ca33096147a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:28.898651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72016d20-d4eb-43aa-9b2f-8ca33096147a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:28.913737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809a17a1-c0f3-4a77-986f-0237a6ef6de4 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:28.950902 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0682846-813f-4743-adae-06ae0532d82f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:28.951885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0682846-813f-4743-adae-06ae0532d82f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:28.952403 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0682846-813f-4743-adae-06ae0532d82f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:28.952874 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0682846-813f-4743-adae-06ae0532d82f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:28.957391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:28.960973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0682846-813f-4743-adae-06ae0532d82f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:28.962695 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0682846-813f-4743-adae-06ae0532d82f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-d0682846-813f-4743-adae-06ae0532d82f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:28.975775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:28.976750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:28.978098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:28.978098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:28.978098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:28.978098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:28.978098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:28.978098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:28.978098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:28.978098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:28.980419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 445/865] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:28.990267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0682846-813f-4743-adae-06ae0532d82f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:29.008474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 421/866] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:28 2022] GET /identity/v3/auth/tokens => generated 1271 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:27:29.011705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:29.014438 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72016d20-d4eb-43aa-9b2f-8ca33096147a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:29.103930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0682846-813f-4743-adae-06ae0532d82f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:29.127214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72016d20-d4eb-43aa-9b2f-8ca33096147a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:29.140601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7142a267-660a-49bf-8c9f-1103f446419c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:29.141204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7142a267-660a-49bf-8c9f-1103f446419c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:29.141484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7142a267-660a-49bf-8c9f-1103f446419c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:29.141754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7142a267-660a-49bf-8c9f-1103f446419c None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:29.142478 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7142a267-660a-49bf-8c9f-1103f446419c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:29.143479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7142a267-660a-49bf-8c9f-1103f446419c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:29.145093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7142a267-660a-49bf-8c9f-1103f446419c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:29.157469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 446/867] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:29.159265 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523dffcc-ddde-4640-b4af-b7741e397841 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:29.159547 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523dffcc-ddde-4640-b4af-b7741e397841 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:29.159818 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523dffcc-ddde-4640-b4af-b7741e397841 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:29.162518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 447/868] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:29.165608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523dffcc-ddde-4640-b4af-b7741e397841 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:29.168745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39674eba-3e47-43a3-b28c-f115131483a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:29.170231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39674eba-3e47-43a3-b28c-f115131483a7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:29.170520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39674eba-3e47-43a3-b28c-f115131483a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:29.170794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39674eba-3e47-43a3-b28c-f115131483a7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:29.182367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39674eba-3e47-43a3-b28c-f115131483a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:29.183798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39674eba-3e47-43a3-b28c-f115131483a7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-39674eba-3e47-43a3-b28c-f115131483a7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:29.198113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:29.199283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:29.200386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:29.200386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:29.200386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:29.203828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 422/869] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:29.205112 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698aee46-544d-404a-8db7-b5d4ce592e07 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:29.205394 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698aee46-544d-404a-8db7-b5d4ce592e07 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:29.206269 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698aee46-544d-404a-8db7-b5d4ce592e07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:29.223427 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-698aee46-544d-404a-8db7-b5d4ce592e07 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:29.224649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-698aee46-544d-404a-8db7-b5d4ce592e07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:29.226694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:29.248393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:29.278891 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523dffcc-ddde-4640-b4af-b7741e397841 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:29.295953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 423/870] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:29 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:29.303038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698aee46-544d-404a-8db7-b5d4ce592e07 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:29.322632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f688f5-40f7-4fb5-9423-5eeb3b89b61f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:29.323288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f688f5-40f7-4fb5-9423-5eeb3b89b61f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:29.323635 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f688f5-40f7-4fb5-9423-5eeb3b89b61f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:29.323834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f688f5-40f7-4fb5-9423-5eeb3b89b61f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:29.324675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f688f5-40f7-4fb5-9423-5eeb3b89b61f None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=4591bec74a4641af818b2d75b657115e) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:29.324983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f688f5-40f7-4fb5-9423-5eeb3b89b61f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:29.326746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f688f5-40f7-4fb5-9423-5eeb3b89b61f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:29.338664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 448/871] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:29 2022] GET /identity/v3/role_assignments?user.id=4591bec74a4641af818b2d75b657115e&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=43ca25467bc6484da3a50508efb79230 => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:29.420685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698aee46-544d-404a-8db7-b5d4ce592e07 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:29.460574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca66725-9c00-43e8-a7a7-b7f3a9c83de2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:29.461120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca66725-9c00-43e8-a7a7-b7f3a9c83de2 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:29.461414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca66725-9c00-43e8-a7a7-b7f3a9c83de2 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:29.461703 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca66725-9c00-43e8-a7a7-b7f3a9c83de2 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:29.462397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca66725-9c00-43e8-a7a7-b7f3a9c83de2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:29.462703 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca66725-9c00-43e8-a7a7-b7f3a9c83de2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:29.464120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca66725-9c00-43e8-a7a7-b7f3a9c83de2 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:29.474564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:29.478338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 424/872] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:29.483762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f688f5-40f7-4fb5-9423-5eeb3b89b61f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:29.568024 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca66725-9c00-43e8-a7a7-b7f3a9c83de2 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:29.613409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f688f5-40f7-4fb5-9423-5eeb3b89b61f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:29.661409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9ebedb-e581-4614-9d55-5a9e7899c128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:29.662041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9ebedb-e581-4614-9d55-5a9e7899c128 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:29.662345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9ebedb-e581-4614-9d55-5a9e7899c128 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:29.662613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9ebedb-e581-4614-9d55-5a9e7899c128 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:29.666460 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:29.672839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9ebedb-e581-4614-9d55-5a9e7899c128 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:29.675149 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9ebedb-e581-4614-9d55-5a9e7899c128 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:29.676040 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca66725-9c00-43e8-a7a7-b7f3a9c83de2 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-bf9ebedb-e581-4614-9d55-5a9e7899c128 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:29.689315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:29.691470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:29.693335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:29.693335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:29.693335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:29.693335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:29.693335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:29.693335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:29.693335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:29.693335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:29.698185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 449/873] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:29.698185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9ebedb-e581-4614-9d55-5a9e7899c128 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:29.724266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3bc2e6-5a08-490e-ac7b-44aa9ea825ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:29.724798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3bc2e6-5a08-490e-ac7b-44aa9ea825ff None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:29.725230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3bc2e6-5a08-490e-ac7b-44aa9ea825ff None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:29.725612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3bc2e6-5a08-490e-ac7b-44aa9ea825ff None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:29.726460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3bc2e6-5a08-490e-ac7b-44aa9ea825ff None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:29.729131 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3bc2e6-5a08-490e-ac7b-44aa9ea825ff None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:29.811332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9ebedb-e581-4614-9d55-5a9e7899c128 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:29.845835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 425/874] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:27:29 2022] POST /identity/v3/services => generated 251 bytes in 278 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:27:29.848614 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1f832e-dca8-4040-8f91-74da094c4a78 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:29.849415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1f832e-dca8-4040-8f91-74da094c4a78 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:29.850640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1f832e-dca8-4040-8f91-74da094c4a78 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:29.852485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 426/875] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:29 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:29.862910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5de8bbe-8c90-4b29-ae77-7d158b758404 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:29.862910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5de8bbe-8c90-4b29-ae77-7d158b758404 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:29.862910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5de8bbe-8c90-4b29-ae77-7d158b758404 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:29.862910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5de8bbe-8c90-4b29-ae77-7d158b758404 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:29.863512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5de8bbe-8c90-4b29-ae77-7d158b758404 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:29.863512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5de8bbe-8c90-4b29-ae77-7d158b758404 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:29.863512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5de8bbe-8c90-4b29-ae77-7d158b758404 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:29.863739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868b1052-442a-44c2-9fe6-29a7ef4569a6 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:29.863739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868b1052-442a-44c2-9fe6-29a7ef4569a6 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:29.863739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868b1052-442a-44c2-9fe6-29a7ef4569a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:29.876567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 450/876] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:29.883464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5de8bbe-8c90-4b29-ae77-7d158b758404 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:29.889849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-868b1052-442a-44c2-9fe6-29a7ef4569a6 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:29.891102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-868b1052-442a-44c2-9fe6-29a7ef4569a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:29.960078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:29.964895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:29.976255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 427/877] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:29 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:30.027262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5de8bbe-8c90-4b29-ae77-7d158b758404 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:30.064434 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cbfab9-a6d5-43e5-be3b-9653e47ba5af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:30.065251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cbfab9-a6d5-43e5-be3b-9653e47ba5af None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:30.065713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cbfab9-a6d5-43e5-be3b-9653e47ba5af None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:30.066081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cbfab9-a6d5-43e5-be3b-9653e47ba5af None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:30.066939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4cbfab9-a6d5-43e5-be3b-9653e47ba5af None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=c63ade06fed946e79dc1e4b5ecfab5e6) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:30.067362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4cbfab9-a6d5-43e5-be3b-9653e47ba5af None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:30.068968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4cbfab9-a6d5-43e5-be3b-9653e47ba5af None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:30.075361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 451/878] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:29 2022] GET /identity/v3/role_assignments?user.id=c63ade06fed946e79dc1e4b5ecfab5e6&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 608 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:30.250635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868b1052-442a-44c2-9fe6-29a7ef4569a6 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:30.359311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868b1052-442a-44c2-9fe6-29a7ef4569a6 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:30.398901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39fd1fe-e44e-48b4-bcc0-64834642cac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:30.399903 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39fd1fe-e44e-48b4-bcc0-64834642cac5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:30.400405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39fd1fe-e44e-48b4-bcc0-64834642cac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:30.400890 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39fd1fe-e44e-48b4-bcc0-64834642cac5 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:30.401669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39fd1fe-e44e-48b4-bcc0-64834642cac5 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:30.403247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39fd1fe-e44e-48b4-bcc0-64834642cac5 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-d39fd1fe-e44e-48b4-bcc0-64834642cac5 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:30.415610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:30.416624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 03:27:30.417528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:30.420349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 428/879] 10.222.0.65 () {56 vars in 1132 bytes} [Wed Aug 24 03:27:30 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:30.428615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cbfab9-a6d5-43e5-be3b-9653e47ba5af None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:30.538411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cbfab9-a6d5-43e5-be3b-9653e47ba5af None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:30.557221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:30.576959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999dbc69-bf66-4865-a646-5ea4e01b43ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:30.577881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999dbc69-bf66-4865-a646-5ea4e01b43ee None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:30.578390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999dbc69-bf66-4865-a646-5ea4e01b43ee None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:30.578880 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999dbc69-bf66-4865-a646-5ea4e01b43ee None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:30.579707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999dbc69-bf66-4865-a646-5ea4e01b43ee None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:30.580202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999dbc69-bf66-4865-a646-5ea4e01b43ee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:30.582090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999dbc69-bf66-4865-a646-5ea4e01b43ee None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:30.587239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 452/880] 10.222.0.65 () {56 vars in 1126 bytes} [Wed Aug 24 03:27:30 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:30.595807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39fd1fe-e44e-48b4-bcc0-64834642cac5 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:30.650596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:30.704099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39fd1fe-e44e-48b4-bcc0-64834642cac5 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:30.740274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cba90d6-ec76-4d5c-bb3b-3178361aa854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:30.741198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cba90d6-ec76-4d5c-bb3b-3178361aa854 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:30.741676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cba90d6-ec76-4d5c-bb3b-3178361aa854 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:30.742191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cba90d6-ec76-4d5c-bb3b-3178361aa854 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:30.743029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cba90d6-ec76-4d5c-bb3b-3178361aa854 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:30.743526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cba90d6-ec76-4d5c-bb3b-3178361aa854 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:30.745083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cba90d6-ec76-4d5c-bb3b-3178361aa854 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:30.750794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 429/881] 10.222.0.65 () {56 vars in 1126 bytes} [Wed Aug 24 03:27:30 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:30.759363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999dbc69-bf66-4865-a646-5ea4e01b43ee None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:30.866005 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999dbc69-bf66-4865-a646-5ea4e01b43ee None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:30.902077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68da55b-6182-48e0-8a18-396fdb7e0ce8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:30.902705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68da55b-6182-48e0-8a18-396fdb7e0ce8 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:30.903045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68da55b-6182-48e0-8a18-396fdb7e0ce8 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:30.903316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68da55b-6182-48e0-8a18-396fdb7e0ce8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:30.904069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68da55b-6182-48e0-8a18-396fdb7e0ce8 None admin] RBAC: Adding query filter params (interface=public, service_id=88db20c4d49249b1bee50092007f0a94, region_id=RegionOne) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:30.904377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68da55b-6182-48e0-8a18-396fdb7e0ce8 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:30.905736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68da55b-6182-48e0-8a18-396fdb7e0ce8 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:30.913728 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 453/882] 10.222.0.65 () {56 vars in 1251 bytes} [Wed Aug 24 03:27:30 2022] GET /identity/v3/endpoints?service_id=88db20c4d49249b1bee50092007f0a94&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:30.919164 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cba90d6-ec76-4d5c-bb3b-3178361aa854 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:31.027521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cba90d6-ec76-4d5c-bb3b-3178361aa854 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:31.044714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:31.063767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227b8b6b-cb17-4fa1-a769-de6e7367e0e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:31.064599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227b8b6b-cb17-4fa1-a769-de6e7367e0e6 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:31.069533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227b8b6b-cb17-4fa1-a769-de6e7367e0e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:31.069533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227b8b6b-cb17-4fa1-a769-de6e7367e0e6 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:31.069533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227b8b6b-cb17-4fa1-a769-de6e7367e0e6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:31.069533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227b8b6b-cb17-4fa1-a769-de6e7367e0e6 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:31.073906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 430/883] 10.222.0.65 () {56 vars in 1087 bytes} [Wed Aug 24 03:27:30 2022] GET /identity/v3/services => generated 1268 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:27:31.390002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fd37c2-6faa-41c0-a661-e4ed04176259 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:31.390402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fd37c2-6faa-41c0-a661-e4ed04176259 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:31.390597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fd37c2-6faa-41c0-a661-e4ed04176259 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:31.392202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 454/884] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:31.397677 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f5163c-6ae0-4b47-9b1b-a0b866a49ceb None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:31.397963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f5163c-6ae0-4b47-9b1b-a0b866a49ceb None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:31.398266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f5163c-6ae0-4b47-9b1b-a0b866a49ceb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:31.425931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-a4f5163c-6ae0-4b47-9b1b-a0b866a49ceb None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:31.427173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a4f5163c-6ae0-4b47-9b1b-a0b866a49ceb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:31.502744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 431/885] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:31 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:31.777259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fd37c2-6faa-41c0-a661-e4ed04176259 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:31.825175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:31.839770 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576ebe3d-6cbb-4dd7-b1d7-ee03cf6a5353 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:31.840275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576ebe3d-6cbb-4dd7-b1d7-ee03cf6a5353 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:31.840364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576ebe3d-6cbb-4dd7-b1d7-ee03cf6a5353 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:31.843254 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 432/886] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:31.848086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea90475d-8365-47bf-9274-cefab63560bd None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:31.848374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea90475d-8365-47bf-9274-cefab63560bd None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:31.848643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea90475d-8365-47bf-9274-cefab63560bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:31.870329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-ea90475d-8365-47bf-9274-cefab63560bd None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:31.871859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ea90475d-8365-47bf-9274-cefab63560bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:31.886739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fd37c2-6faa-41c0-a661-e4ed04176259 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:31.921782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ddd57b7-ec42-4302-9ea1-9259c1eaa2ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:31.922452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddd57b7-ec42-4302-9ea1-9259c1eaa2ec None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:31.922714 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddd57b7-ec42-4302-9ea1-9259c1eaa2ec None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:31.922986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ddd57b7-ec42-4302-9ea1-9259c1eaa2ec None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:31.923565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ddd57b7-ec42-4302-9ea1-9259c1eaa2ec None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:31.925629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ddd57b7-ec42-4302-9ea1-9259c1eaa2ec None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-7ddd57b7-ec42-4302-9ea1-9259c1eaa2ec None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:31.934560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:31.935588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:31.936433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 455/887] 10.222.0.65 () {56 vars in 1126 bytes} [Wed Aug 24 03:27:31 2022] GET /identity/v3/services/object-store => generated 93 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:31.944147 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ddd57b7-ec42-4302-9ea1-9259c1eaa2ec None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:31.959888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 433/888] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:31 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:32.052439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ddd57b7-ec42-4302-9ea1-9259c1eaa2ec None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:32.062537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:32.088842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b8ef79-e8fa-40ae-95e5-9cb2682d0381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:32.089738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b8ef79-e8fa-40ae-95e5-9cb2682d0381 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:32.090267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b8ef79-e8fa-40ae-95e5-9cb2682d0381 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:32.090732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b8ef79-e8fa-40ae-95e5-9cb2682d0381 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:32.091555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b8ef79-e8fa-40ae-95e5-9cb2682d0381 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:32.092033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b8ef79-e8fa-40ae-95e5-9cb2682d0381 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:32.093638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b8ef79-e8fa-40ae-95e5-9cb2682d0381 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:32.099772 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 456/889] 10.222.0.65 () {56 vars in 1122 bytes} [Wed Aug 24 03:27:31 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:32.110196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea90475d-8365-47bf-9274-cefab63560bd None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:32.149339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:32.213835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea90475d-8365-47bf-9274-cefab63560bd None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:32.235411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b8ef79-e8fa-40ae-95e5-9cb2682d0381 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:32.250573 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e2bd10-9dca-4da8-8503-8ce10cfe60a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:32.251178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e2bd10-9dca-4da8-8503-8ce10cfe60a3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:32.251456 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e2bd10-9dca-4da8-8503-8ce10cfe60a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:32.251726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e2bd10-9dca-4da8-8503-8ce10cfe60a3 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:32.252493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e2bd10-9dca-4da8-8503-8ce10cfe60a3 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:32.252951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e2bd10-9dca-4da8-8503-8ce10cfe60a3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:32.254342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e2bd10-9dca-4da8-8503-8ce10cfe60a3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:32.259660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 434/890] 10.222.0.65 () {56 vars in 1122 bytes} [Wed Aug 24 03:27:32 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:32.288121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc770e8b-2237-455b-b5ad-3ba95b533e98 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:32.288496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc770e8b-2237-455b-b5ad-3ba95b533e98 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:32.288682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc770e8b-2237-455b-b5ad-3ba95b533e98 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:32.290363 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 435/891] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:32.296241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad24762-a569-4782-ab3b-c75afe78fd9d None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:32.296466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad24762-a569-4782-ab3b-c75afe78fd9d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:32.296673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad24762-a569-4782-ab3b-c75afe78fd9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:32.316066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-0ad24762-a569-4782-ab3b-c75afe78fd9d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:32.317341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0ad24762-a569-4782-ab3b-c75afe78fd9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:32.354669 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b8ef79-e8fa-40ae-95e5-9cb2682d0381 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:32.394311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 436/892] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:32 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:32.398026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45747b86-e30c-425f-b78c-a258c6cbaa13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:32.399059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45747b86-e30c-425f-b78c-a258c6cbaa13 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:32.399539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45747b86-e30c-425f-b78c-a258c6cbaa13 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:32.400006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45747b86-e30c-425f-b78c-a258c6cbaa13 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:32.400831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45747b86-e30c-425f-b78c-a258c6cbaa13 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:32.402451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45747b86-e30c-425f-b78c-a258c6cbaa13 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-45747b86-e30c-425f-b78c-a258c6cbaa13 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:32.421828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:32.422927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:32.423782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 457/893] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:27:32 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:32.424710 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad24762-a569-4782-ab3b-c75afe78fd9d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:32.478522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:32.536466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad24762-a569-4782-ab3b-c75afe78fd9d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:32.573085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3764f011-21a5-4b52-b9b2-8438cac48248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:32.574066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3764f011-21a5-4b52-b9b2-8438cac48248 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:32.574622 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3764f011-21a5-4b52-b9b2-8438cac48248 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:32.575097 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3764f011-21a5-4b52-b9b2-8438cac48248 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:32.575967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3764f011-21a5-4b52-b9b2-8438cac48248 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:32.576489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3764f011-21a5-4b52-b9b2-8438cac48248 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:32.578100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3764f011-21a5-4b52-b9b2-8438cac48248 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:32.583810 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 437/894] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:27:32 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:32.593238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45747b86-e30c-425f-b78c-a258c6cbaa13 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:32.627084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:32.680374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3764f011-21a5-4b52-b9b2-8438cac48248 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:32.696469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45747b86-e30c-425f-b78c-a258c6cbaa13 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:32.732707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a47226-e069-489b-a814-55a4f304ce1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:32.733325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a47226-e069-489b-a814-55a4f304ce1f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:32.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a47226-e069-489b-a814-55a4f304ce1f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:32.733874 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a47226-e069-489b-a814-55a4f304ce1f None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:32.734646 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a47226-e069-489b-a814-55a4f304ce1f None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:32.734952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a47226-e069-489b-a814-55a4f304ce1f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:32.736428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a47226-e069-489b-a814-55a4f304ce1f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:32.742209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 458/895] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:27:32 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:32.748011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a47226-e069-489b-a814-55a4f304ce1f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:32.794837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:32.799268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3764f011-21a5-4b52-b9b2-8438cac48248 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:32.835276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b51061-493b-425f-90f3-3cf26b83d82d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:32.836577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b51061-493b-425f-90f3-3cf26b83d82d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:32.836856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b51061-493b-425f-90f3-3cf26b83d82d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:32.837124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b51061-493b-425f-90f3-3cf26b83d82d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:32.851627 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b51061-493b-425f-90f3-3cf26b83d82d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:32.859722 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-b7b51061-493b-425f-90f3-3cf26b83d82d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:32.861167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:32.862388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:32.863405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:32.863405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:32.863405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:32.863405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:32.863405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:32.863405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:32.863405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:32.863405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:32.863405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:32.863405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:32.863405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 03:27:32.863405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:32.864923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a47226-e069-489b-a814-55a4f304ce1f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:32.870923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 438/896] 10.222.0.65 () {56 vars in 1120 bytes} [Wed Aug 24 03:27:32 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:32.872902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b51061-493b-425f-90f3-3cf26b83d82d None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:32.908402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45452623-3178-4510-b658-54f96586bb4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:32.909338 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45452623-3178-4510-b658-54f96586bb4a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:32.909789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45452623-3178-4510-b658-54f96586bb4a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:32.910246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45452623-3178-4510-b658-54f96586bb4a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:32.911127 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45452623-3178-4510-b658-54f96586bb4a None admin] RBAC: Adding query filter params (interface=public, service_id=2c2185660a2948fd8114aea115e93134, region_id=RegionOne) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:32.911578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45452623-3178-4510-b658-54f96586bb4a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:32.913095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45452623-3178-4510-b658-54f96586bb4a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:32.918936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 459/897] 10.222.0.65 () {56 vars in 1251 bytes} [Wed Aug 24 03:27:32 2022] GET /identity/v3/endpoints?service_id=2c2185660a2948fd8114aea115e93134&interface=public®ion_id=RegionOne => generated 194 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:32.924988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45452623-3178-4510-b658-54f96586bb4a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:32.991751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b51061-493b-425f-90f3-3cf26b83d82d None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:33.029346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86b3ebf-b0c7-4565-a260-c97261efa472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:33.030094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86b3ebf-b0c7-4565-a260-c97261efa472 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:33.030509 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86b3ebf-b0c7-4565-a260-c97261efa472 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:33.030882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86b3ebf-b0c7-4565-a260-c97261efa472 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:33.031331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45452623-3178-4510-b658-54f96586bb4a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:33.031739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86b3ebf-b0c7-4565-a260-c97261efa472 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:33.032138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86b3ebf-b0c7-4565-a260-c97261efa472 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:33.033751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86b3ebf-b0c7-4565-a260-c97261efa472 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:33.046577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 439/898] 10.222.0.65 () {56 vars in 1115 bytes} [Wed Aug 24 03:27:32 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:33.048851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86b3ebf-b0c7-4565-a260-c97261efa472 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:33.068873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83efd734-c927-4c71-99c1-c53e5f7007ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:33.069502 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83efd734-c927-4c71-99c1-c53e5f7007ee None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:33.069895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83efd734-c927-4c71-99c1-c53e5f7007ee None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:33.070198 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83efd734-c927-4c71-99c1-c53e5f7007ee None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:33.070831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83efd734-c927-4c71-99c1-c53e5f7007ee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:33.072195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83efd734-c927-4c71-99c1-c53e5f7007ee None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:33.074450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:33.078457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 460/899] 10.222.0.65 () {56 vars in 1087 bytes} [Wed Aug 24 03:27:32 2022] GET /identity/v3/services => generated 1268 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:27:33.079694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95e0a68-fa89-4bd6-850c-dc73ab291f8b None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:33.079950 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95e0a68-fa89-4bd6-850c-dc73ab291f8b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:33.080224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95e0a68-fa89-4bd6-850c-dc73ab291f8b None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:33.082189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 461/900] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:33.093001 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c5841e-ff2e-4296-9554-1252fd3e70e5 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:33.093001 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c5841e-ff2e-4296-9554-1252fd3e70e5 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:33.093001 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c5841e-ff2e-4296-9554-1252fd3e70e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:33.115035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-89c5841e-ff2e-4296-9554-1252fd3e70e5 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:33.117029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-89c5841e-ff2e-4296-9554-1252fd3e70e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:33.151335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:33.181007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86b3ebf-b0c7-4565-a260-c97261efa472 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:33.206605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 462/901] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:33 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:33.221985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8197de32-1845-4f6d-94e6-db2d3684b7be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:33.223043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8197de32-1845-4f6d-94e6-db2d3684b7be None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:33.223576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8197de32-1845-4f6d-94e6-db2d3684b7be None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:33.224112 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8197de32-1845-4f6d-94e6-db2d3684b7be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:33.235443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8197de32-1845-4f6d-94e6-db2d3684b7be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:33.237425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8197de32-1845-4f6d-94e6-db2d3684b7be None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-8197de32-1845-4f6d-94e6-db2d3684b7be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:33.254073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:33.255115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:33.256305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:33.256305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:33.256305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:33.256305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 440/902] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:33.262980 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c5841e-ff2e-4296-9554-1252fd3e70e5 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:33.366353 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c5841e-ff2e-4296-9554-1252fd3e70e5 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:33.402046 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7abf0ca-e30a-4eaf-8d0d-32c707678981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:33.402706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7abf0ca-e30a-4eaf-8d0d-32c707678981 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:33.402985 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7abf0ca-e30a-4eaf-8d0d-32c707678981 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:33.403260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7abf0ca-e30a-4eaf-8d0d-32c707678981 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:33.403926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7abf0ca-e30a-4eaf-8d0d-32c707678981 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:33.404223 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7abf0ca-e30a-4eaf-8d0d-32c707678981 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:33.405653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7abf0ca-e30a-4eaf-8d0d-32c707678981 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:33.419826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 463/903] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:33.427238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8197de32-1845-4f6d-94e6-db2d3684b7be None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:33.504187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:33.532275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7abf0ca-e30a-4eaf-8d0d-32c707678981 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:33.535862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8197de32-1845-4f6d-94e6-db2d3684b7be None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:33.553591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:33.574063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9507d76-0809-4a14-a1ab-0557f792a1f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:33.574729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9507d76-0809-4a14-a1ab-0557f792a1f9 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:33.575063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9507d76-0809-4a14-a1ab-0557f792a1f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:33.575335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9507d76-0809-4a14-a1ab-0557f792a1f9 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:33.585063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9507d76-0809-4a14-a1ab-0557f792a1f9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:33.586525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9507d76-0809-4a14-a1ab-0557f792a1f9 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-a9507d76-0809-4a14-a1ab-0557f792a1f9 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:33.599496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:33.600654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:33.602928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:33.602928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:33.602928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:33.602928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:33.602928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:33.602928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:33.602928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:33.602928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:33.602928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 03:27:33.602928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:33.602928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 441/904] 10.222.0.65 () {56 vars in 1117 bytes} [Wed Aug 24 03:27:33 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:33.610175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9507d76-0809-4a14-a1ab-0557f792a1f9 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:33.643892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7abf0ca-e30a-4eaf-8d0d-32c707678981 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:33.679776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10acb13-6094-4c4d-92e6-aa697bfafb9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:33.680500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10acb13-6094-4c4d-92e6-aa697bfafb9d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:33.680785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10acb13-6094-4c4d-92e6-aa697bfafb9d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:33.681063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10acb13-6094-4c4d-92e6-aa697bfafb9d None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:33.681746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10acb13-6094-4c4d-92e6-aa697bfafb9d None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:33.683176 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10acb13-6094-4c4d-92e6-aa697bfafb9d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-b10acb13-6094-4c4d-92e6-aa697bfafb9d None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:33.692204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:33.694427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 24 03:27:33.696538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:33.702501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 464/905] 10.222.0.65 () {56 vars in 1132 bytes} [Wed Aug 24 03:27:33 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:33.702501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10acb13-6094-4c4d-92e6-aa697bfafb9d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:33.716487 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9507d76-0809-4a14-a1ab-0557f792a1f9 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:33.748265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:33.754480 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95313017-0a0e-4fde-94e5-8489554b6251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:33.755084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95313017-0a0e-4fde-94e5-8489554b6251 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:33.755352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95313017-0a0e-4fde-94e5-8489554b6251 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:33.755873 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95313017-0a0e-4fde-94e5-8489554b6251 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:33.756617 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95313017-0a0e-4fde-94e5-8489554b6251 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:33.757095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95313017-0a0e-4fde-94e5-8489554b6251 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:33.758704 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95313017-0a0e-4fde-94e5-8489554b6251 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:33.783016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 442/906] 10.222.0.65 () {56 vars in 1149 bytes} [Wed Aug 24 03:27:33 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:33.785016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95313017-0a0e-4fde-94e5-8489554b6251 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:33.793971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:33.813838 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10acb13-6094-4c4d-92e6-aa697bfafb9d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:33.850331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdafa08-a6c2-4159-8f91-c40d5df5f0b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:33.850987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdafa08-a6c2-4159-8f91-c40d5df5f0b8 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:33.851445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdafa08-a6c2-4159-8f91-c40d5df5f0b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:33.851590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdafa08-a6c2-4159-8f91-c40d5df5f0b8 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:33.852154 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdafa08-a6c2-4159-8f91-c40d5df5f0b8 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:33.852535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdafa08-a6c2-4159-8f91-c40d5df5f0b8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:33.853839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdafa08-a6c2-4159-8f91-c40d5df5f0b8 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:33.859014 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 465/907] 10.222.0.65 () {56 vars in 1126 bytes} [Wed Aug 24 03:27:33 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:33.865553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdafa08-a6c2-4159-8f91-c40d5df5f0b8 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:33.898263 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95313017-0a0e-4fde-94e5-8489554b6251 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:33.936185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92290288-a914-4ca8-9728-fb046a8a5846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:33.937035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92290288-a914-4ca8-9728-fb046a8a5846 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:33.937459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92290288-a914-4ca8-9728-fb046a8a5846 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:33.937991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92290288-a914-4ca8-9728-fb046a8a5846 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:33.971639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdafa08-a6c2-4159-8f91-c40d5df5f0b8 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:34.001635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:34.007107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18b398b-e7ed-4b8b-95ad-eefdd1f7a0e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:34.007620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18b398b-e7ed-4b8b-95ad-eefdd1f7a0e9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:34.007905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18b398b-e7ed-4b8b-95ad-eefdd1f7a0e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:34.008318 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18b398b-e7ed-4b8b-95ad-eefdd1f7a0e9 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:34.008881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18b398b-e7ed-4b8b-95ad-eefdd1f7a0e9 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:34.009177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18b398b-e7ed-4b8b-95ad-eefdd1f7a0e9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:34.010644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18b398b-e7ed-4b8b-95ad-eefdd1f7a0e9 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:34.015054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92290288-a914-4ca8-9728-fb046a8a5846 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:34.016670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92290288-a914-4ca8-9728-fb046a8a5846 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:34.021431 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 466/908] 10.222.0.65 () {56 vars in 1126 bytes} [Wed Aug 24 03:27:33 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:34.022356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18b398b-e7ed-4b8b-95ad-eefdd1f7a0e9 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:34.035658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:34.132857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18b398b-e7ed-4b8b-95ad-eefdd1f7a0e9 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:34.133354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 443/909] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:33 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:27:34.134442 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92290288-a914-4ca8-9728-fb046a8a5846 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:34.165338 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3355f9b4-9077-4f0e-9cf4-44e7ce7ecde1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:34.165927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3355f9b4-9077-4f0e-9cf4-44e7ce7ecde1 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:34.166233 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3355f9b4-9077-4f0e-9cf4-44e7ce7ecde1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:34.166510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3355f9b4-9077-4f0e-9cf4-44e7ce7ecde1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:34.167189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3355f9b4-9077-4f0e-9cf4-44e7ce7ecde1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:34.168702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3355f9b4-9077-4f0e-9cf4-44e7ce7ecde1 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:34.232555 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:34.242418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92290288-a914-4ca8-9728-fb046a8a5846 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:34.261642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 467/910] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:27:34 2022] POST /identity/v3/endpoints => generated 347 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:34.270797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29714e1e-0a77-4723-b2a1-c168bfe621ed None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:34.271081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29714e1e-0a77-4723-b2a1-c168bfe621ed None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:34.271361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29714e1e-0a77-4723-b2a1-c168bfe621ed None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:34.272937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 468/911] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:34.278606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55799a12-548a-48a6-bd54-01a6bff68e9b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:34.278889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55799a12-548a-48a6-bd54-01a6bff68e9b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:34.279157 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55799a12-548a-48a6-bd54-01a6bff68e9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:34.292921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc98f41-c043-4e4e-baa5-c8aa49b47c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:34.293826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc98f41-c043-4e4e-baa5-c8aa49b47c40 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:34.294297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc98f41-c043-4e4e-baa5-c8aa49b47c40 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:34.295057 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc98f41-c043-4e4e-baa5-c8aa49b47c40 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:34.309439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc98f41-c043-4e4e-baa5-c8aa49b47c40 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:34.311298 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc98f41-c043-4e4e-baa5-c8aa49b47c40 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:34.311416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-55799a12-548a-48a6-bd54-01a6bff68e9b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:34.312929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-55799a12-548a-48a6-bd54-01a6bff68e9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-0dc98f41-c043-4e4e-baa5-c8aa49b47c40 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:34.332838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:34.334008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:34.334947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:34.334947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:34.334947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:34.338519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 444/912] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:34.341140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc98f41-c043-4e4e-baa5-c8aa49b47c40 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:34.417053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:34.429390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 469/913] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:34 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:34.475834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc98f41-c043-4e4e-baa5-c8aa49b47c40 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:34.512652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fa8e52-4015-4a0b-9c66-cb26d921cb49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:34.513263 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fa8e52-4015-4a0b-9c66-cb26d921cb49 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:34.513541 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fa8e52-4015-4a0b-9c66-cb26d921cb49 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:34.513874 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fa8e52-4015-4a0b-9c66-cb26d921cb49 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:34.514556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fa8e52-4015-4a0b-9c66-cb26d921cb49 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:34.514906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fa8e52-4015-4a0b-9c66-cb26d921cb49 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:34.516273 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fa8e52-4015-4a0b-9c66-cb26d921cb49 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:34.524966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:34.530615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 445/914] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:34.536367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55799a12-548a-48a6-bd54-01a6bff68e9b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:34.648222 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55799a12-548a-48a6-bd54-01a6bff68e9b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:34.685015 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca5e63-3351-434b-9a65-6e6380aaf474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:34.685569 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca5e63-3351-434b-9a65-6e6380aaf474 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:34.685871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca5e63-3351-434b-9a65-6e6380aaf474 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:34.686143 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca5e63-3351-434b-9a65-6e6380aaf474 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:34.693251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ca5e63-3351-434b-9a65-6e6380aaf474 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:34.694734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ca5e63-3351-434b-9a65-6e6380aaf474 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-e8ca5e63-3351-434b-9a65-6e6380aaf474 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:34.706730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:34.707650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:34.708540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:34.708540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:34.708540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:34.708540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:34.708540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:34.708540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:34.708540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:34.708540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:34.708540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 470/915] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:34.710849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fa8e52-4015-4a0b-9c66-cb26d921cb49 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:34.715029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca5e63-3351-434b-9a65-6e6380aaf474 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:34.776479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:34.827661 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca5e63-3351-434b-9a65-6e6380aaf474 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:34.865803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cae8ad2-af2d-49aa-bdb1-c7400dffbf50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:34.866889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cae8ad2-af2d-49aa-bdb1-c7400dffbf50 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:34.867008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cae8ad2-af2d-49aa-bdb1-c7400dffbf50 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:34.867191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cae8ad2-af2d-49aa-bdb1-c7400dffbf50 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:34.867893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cae8ad2-af2d-49aa-bdb1-c7400dffbf50 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:34.868242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cae8ad2-af2d-49aa-bdb1-c7400dffbf50 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:34.869682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cae8ad2-af2d-49aa-bdb1-c7400dffbf50 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:34.870809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fa8e52-4015-4a0b-9c66-cb26d921cb49 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:34.888762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 471/916] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:34.889695 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cae8ad2-af2d-49aa-bdb1-c7400dffbf50 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:34.918839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4168ccf8-6c0e-41ff-b186-ed9d16d2361f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:34.920737 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4168ccf8-6c0e-41ff-b186-ed9d16d2361f None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:34.921174 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4168ccf8-6c0e-41ff-b186-ed9d16d2361f None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:34.921587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4168ccf8-6c0e-41ff-b186-ed9d16d2361f None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:34.922696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4168ccf8-6c0e-41ff-b186-ed9d16d2361f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:34.924893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4168ccf8-6c0e-41ff-b186-ed9d16d2361f None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:34.998867 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cae8ad2-af2d-49aa-bdb1-c7400dffbf50 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:35.009618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 446/917] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:27:34 2022] POST /identity/v3/services => generated 241 bytes in 299 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:35.030833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:35.060788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c633c09e-ae3e-48f6-862c-13faee3f0091 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:35.061421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c633c09e-ae3e-48f6-862c-13faee3f0091 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:35.061696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c633c09e-ae3e-48f6-862c-13faee3f0091 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:35.061962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c633c09e-ae3e-48f6-862c-13faee3f0091 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:35.062934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c633c09e-ae3e-48f6-862c-13faee3f0091 None admin] RBAC: Adding query filter params (role.id=1489a00dd22146ecb6161ff7d549f95c, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=c63ade06fed946e79dc1e4b5ecfab5e6) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:35.063247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c633c09e-ae3e-48f6-862c-13faee3f0091 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:35.064860 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c633c09e-ae3e-48f6-862c-13faee3f0091 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:35.071800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 472/918] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:34 2022] GET /identity/v3/role_assignments?user.id=c63ade06fed946e79dc1e4b5ecfab5e6&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=1489a00dd22146ecb6161ff7d549f95c => generated 259 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:35.112313 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfebaf9b-dab8-4387-b467-35d941b47916 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:35.112604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfebaf9b-dab8-4387-b467-35d941b47916 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:35.112841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfebaf9b-dab8-4387-b467-35d941b47916 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:35.114654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 447/919] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:35.122005 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e6b1ef-9432-46ef-9ade-2155637913eb None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:35.122399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e6b1ef-9432-46ef-9ade-2155637913eb None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:35.122574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e6b1ef-9432-46ef-9ade-2155637913eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:35.144653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-46e6b1ef-9432-46ef-9ade-2155637913eb None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:35.145941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-46e6b1ef-9432-46ef-9ade-2155637913eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:35.223358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 473/920] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:35.495579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfebaf9b-dab8-4387-b467-35d941b47916 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:35.560993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:35.610405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfebaf9b-dab8-4387-b467-35d941b47916 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:35.646389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309ca1aa-e16f-4454-85b8-cb02dacb73ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:35.647018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309ca1aa-e16f-4454-85b8-cb02dacb73ca None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:35.647296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309ca1aa-e16f-4454-85b8-cb02dacb73ca None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:35.647566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309ca1aa-e16f-4454-85b8-cb02dacb73ca None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:35.648143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309ca1aa-e16f-4454-85b8-cb02dacb73ca None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:35.649501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309ca1aa-e16f-4454-85b8-cb02dacb73ca None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-309ca1aa-e16f-4454-85b8-cb02dacb73ca None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:35.658998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:35.660132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:35.661176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 448/921] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:27:35 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:35.668756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e6b1ef-9432-46ef-9ade-2155637913eb None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:35.768610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e6b1ef-9432-46ef-9ade-2155637913eb None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:35.776193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:35.803440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d55a21-0e6e-4e9f-b183-dd5b531e9b4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:35.804052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d55a21-0e6e-4e9f-b183-dd5b531e9b4d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:35.804328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d55a21-0e6e-4e9f-b183-dd5b531e9b4d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:35.804598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d55a21-0e6e-4e9f-b183-dd5b531e9b4d None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:35.805223 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d55a21-0e6e-4e9f-b183-dd5b531e9b4d None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:35.805572 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d55a21-0e6e-4e9f-b183-dd5b531e9b4d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:35.807097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d55a21-0e6e-4e9f-b183-dd5b531e9b4d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:35.815592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 474/922] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:27:35 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:35.818567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309ca1aa-e16f-4454-85b8-cb02dacb73ca None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:35.921746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309ca1aa-e16f-4454-85b8-cb02dacb73ca None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:35.948312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:35.960691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c0f89e-1d73-4783-8fae-00b1bf53226d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:35.962533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c0f89e-1d73-4783-8fae-00b1bf53226d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:35.962533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c0f89e-1d73-4783-8fae-00b1bf53226d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:35.962533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c0f89e-1d73-4783-8fae-00b1bf53226d None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:35.963319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c0f89e-1d73-4783-8fae-00b1bf53226d None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:35.963845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c0f89e-1d73-4783-8fae-00b1bf53226d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:35.965416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c0f89e-1d73-4783-8fae-00b1bf53226d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:35.970928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 449/923] 10.222.0.65 () {56 vars in 1114 bytes} [Wed Aug 24 03:27:35 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:35.976878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d55a21-0e6e-4e9f-b183-dd5b531e9b4d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:36.084726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d55a21-0e6e-4e9f-b183-dd5b531e9b4d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:36.120817 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86458dc-99cb-4758-855c-748f14c5dac6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:36.121428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86458dc-99cb-4758-855c-748f14c5dac6 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:36.121713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86458dc-99cb-4758-855c-748f14c5dac6 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:36.121986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86458dc-99cb-4758-855c-748f14c5dac6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:36.122701 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86458dc-99cb-4758-855c-748f14c5dac6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:36.124059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86458dc-99cb-4758-855c-748f14c5dac6 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:36.244908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 475/924] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:27:35 2022] POST /identity/v3/endpoints => generated 346 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:36.421519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5d3c49-ed1d-4a50-8d95-5674deb6ab69 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:36.421793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5d3c49-ed1d-4a50-8d95-5674deb6ab69 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:36.422069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5d3c49-ed1d-4a50-8d95-5674deb6ab69 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:36.423580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 450/925] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:36.428847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d6e513-9117-47ea-a1a7-8c785202b971 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:36.429433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d6e513-9117-47ea-a1a7-8c785202b971 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:36.429927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d6e513-9117-47ea-a1a7-8c785202b971 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:36.450292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e9d6e513-9117-47ea-a1a7-8c785202b971 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:36.451857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e9d6e513-9117-47ea-a1a7-8c785202b971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:36.474685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:36.530504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 476/926] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:36 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:36.793988 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5d3c49-ed1d-4a50-8d95-5674deb6ab69 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:36.897556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5d3c49-ed1d-4a50-8d95-5674deb6ab69 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:36.942187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604d7f99-0d5a-4427-bbf0-3c7af02bb3d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:36.942843 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604d7f99-0d5a-4427-bbf0-3c7af02bb3d5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:36.942843 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604d7f99-0d5a-4427-bbf0-3c7af02bb3d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:36.943000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604d7f99-0d5a-4427-bbf0-3c7af02bb3d5 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:36.946173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604d7f99-0d5a-4427-bbf0-3c7af02bb3d5 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:36.946173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604d7f99-0d5a-4427-bbf0-3c7af02bb3d5 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-604d7f99-0d5a-4427-bbf0-3c7af02bb3d5 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:36.957911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:36.958979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:36.959819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 451/927] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:36 2022] GET /identity/v3/services/compute => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:36.961919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d6e513-9117-47ea-a1a7-8c785202b971 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:37.031393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd2892a-7539-4203-b33a-010b329486fe None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:37.031723 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd2892a-7539-4203-b33a-010b329486fe None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:37.031912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd2892a-7539-4203-b33a-010b329486fe None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:37.033578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 452/928] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:37.038662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb6a3d5-591b-400f-a29b-86c071e5e2ed None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:37.038941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb6a3d5-591b-400f-a29b-86c071e5e2ed None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:37.039212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb6a3d5-591b-400f-a29b-86c071e5e2ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:37.058365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-7cb6a3d5-591b-400f-a29b-86c071e5e2ed None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:37.059687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7cb6a3d5-591b-400f-a29b-86c071e5e2ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:37.071337 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d6e513-9117-47ea-a1a7-8c785202b971 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:37.114753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:37.115456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8e1f58-3847-46b6-896b-ccf320901732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:37.116277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8e1f58-3847-46b6-896b-ccf320901732 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:37.116559 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8e1f58-3847-46b6-896b-ccf320901732 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:37.116832 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8e1f58-3847-46b6-896b-ccf320901732 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:37.117458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8e1f58-3847-46b6-896b-ccf320901732 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:37.117760 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8e1f58-3847-46b6-896b-ccf320901732 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:37.119151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8e1f58-3847-46b6-896b-ccf320901732 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:37.129117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 477/929] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:36 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:37.129117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e4fd98-c6c2-4268-b1d6-8902bdc807c2 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:37.129117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e4fd98-c6c2-4268-b1d6-8902bdc807c2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:37.129117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e4fd98-c6c2-4268-b1d6-8902bdc807c2 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:37.129117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 478/930] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:37.132919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e4fd98-c6c2-4268-b1d6-8902bdc807c2 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:37.145822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 453/931] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:37 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:37.147015 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad17013-34bf-4ae0-ac83-9d66c64ca348 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:37.147374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad17013-34bf-4ae0-ac83-9d66c64ca348 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:37.147721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad17013-34bf-4ae0-ac83-9d66c64ca348 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:37.168320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-cad17013-34bf-4ae0-ac83-9d66c64ca348 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:37.168320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-cad17013-34bf-4ae0-ac83-9d66c64ca348 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:37.173594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:37.250573 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 454/932] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:37 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:37.280247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e4fd98-c6c2-4268-b1d6-8902bdc807c2 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:37.318220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c25de1f-1220-4d8d-b98d-f25e7750968b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:37.318754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c25de1f-1220-4d8d-b98d-f25e7750968b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:37.319034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c25de1f-1220-4d8d-b98d-f25e7750968b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:37.319356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c25de1f-1220-4d8d-b98d-f25e7750968b None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:37.319997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c25de1f-1220-4d8d-b98d-f25e7750968b None admin] RBAC: Adding query filter params (type=compute) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:37.320296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c25de1f-1220-4d8d-b98d-f25e7750968b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:37.321652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c25de1f-1220-4d8d-b98d-f25e7750968b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:37.326970 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 479/933] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:37 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:37.433686 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad17013-34bf-4ae0-ac83-9d66c64ca348 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:37.521894 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c25de1f-1220-4d8d-b98d-f25e7750968b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:37.551510 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad17013-34bf-4ae0-ac83-9d66c64ca348 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:37.563696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:37.590264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bac921b-ea55-4707-aa84-1b42f9ab6366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:37.590781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bac921b-ea55-4707-aa84-1b42f9ab6366 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:37.591059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bac921b-ea55-4707-aa84-1b42f9ab6366 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:37.591330 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bac921b-ea55-4707-aa84-1b42f9ab6366 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:37.603405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bac921b-ea55-4707-aa84-1b42f9ab6366 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-6bac921b-ea55-4707-aa84-1b42f9ab6366 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:37.608608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:37.609580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:37.610465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 455/934] 10.222.0.65 () {56 vars in 1120 bytes} [Wed Aug 24 03:27:37 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:37.616099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bac921b-ea55-4707-aa84-1b42f9ab6366 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:37.635317 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:37.656416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c25de1f-1220-4d8d-b98d-f25e7750968b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:37.697262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf546bc-723a-4448-b41d-25f153158bc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:37.697895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf546bc-723a-4448-b41d-25f153158bc8 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:37.698193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf546bc-723a-4448-b41d-25f153158bc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:37.698532 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf546bc-723a-4448-b41d-25f153158bc8 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:37.699051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf546bc-723a-4448-b41d-25f153158bc8 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:37.700454 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf546bc-723a-4448-b41d-25f153158bc8 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-0bf546bc-723a-4448-b41d-25f153158bc8 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:37.710467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:37.711497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:37.712415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 480/935] 10.222.0.65 () {56 vars in 1126 bytes} [Wed Aug 24 03:27:37 2022] GET /identity/v3/services/object-store => generated 93 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:37.717855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf546bc-723a-4448-b41d-25f153158bc8 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:37.765592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bac921b-ea55-4707-aa84-1b42f9ab6366 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:37.812826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10113f-3821-4d49-958f-6895cc0ff242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:37.813827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10113f-3821-4d49-958f-6895cc0ff242 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:37.815445 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10113f-3821-4d49-958f-6895cc0ff242 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:37.815938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10113f-3821-4d49-958f-6895cc0ff242 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:37.817197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd10113f-3821-4d49-958f-6895cc0ff242 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:37.817969 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd10113f-3821-4d49-958f-6895cc0ff242 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:37.820298 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd10113f-3821-4d49-958f-6895cc0ff242 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:37.827429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf546bc-723a-4448-b41d-25f153158bc8 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:37.837396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 456/936] 10.222.0.65 () {56 vars in 1115 bytes} [Wed Aug 24 03:27:37 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:37.838574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10113f-3821-4d49-958f-6895cc0ff242 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:37.872138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902b83c1-40f6-497f-b267-b0c8563e2deb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:37.872934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902b83c1-40f6-497f-b267-b0c8563e2deb None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:37.873210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902b83c1-40f6-497f-b267-b0c8563e2deb None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:37.873480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902b83c1-40f6-497f-b267-b0c8563e2deb None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:37.874117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902b83c1-40f6-497f-b267-b0c8563e2deb None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:37.875582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902b83c1-40f6-497f-b267-b0c8563e2deb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:37.876955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902b83c1-40f6-497f-b267-b0c8563e2deb None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:37.879173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:37.886495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 481/937] 10.222.0.65 () {56 vars in 1122 bytes} [Wed Aug 24 03:27:37 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:37.892289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902b83c1-40f6-497f-b267-b0c8563e2deb None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:37.905994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:37.967226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10113f-3821-4d49-958f-6895cc0ff242 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:38.007054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c352839-73cd-48c6-840b-c0e07cf5fbc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:38.008014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c352839-73cd-48c6-840b-c0e07cf5fbc7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:38.008436 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c352839-73cd-48c6-840b-c0e07cf5fbc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:38.008826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c352839-73cd-48c6-840b-c0e07cf5fbc7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:38.014200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902b83c1-40f6-497f-b267-b0c8563e2deb None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:38.025124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c352839-73cd-48c6-840b-c0e07cf5fbc7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:38.027084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c352839-73cd-48c6-840b-c0e07cf5fbc7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-8c352839-73cd-48c6-840b-c0e07cf5fbc7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:38.046870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:38.048076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:38.049259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:38.049259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:38.049259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:38.053648 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 457/938] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:38.055132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c352839-73cd-48c6-840b-c0e07cf5fbc7 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:38.060097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1aa826-6c19-43e0-8565-d357549c44ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:38.060708 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1aa826-6c19-43e0-8565-d357549c44ec None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:38.060988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1aa826-6c19-43e0-8565-d357549c44ec None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:38.061261 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1aa826-6c19-43e0-8565-d357549c44ec None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:38.061896 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1aa826-6c19-43e0-8565-d357549c44ec None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:38.062214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1aa826-6c19-43e0-8565-d357549c44ec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:38.063578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1aa826-6c19-43e0-8565-d357549c44ec None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:38.071454 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 482/939] 10.222.0.65 () {56 vars in 1122 bytes} [Wed Aug 24 03:27:37 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:38.077138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1aa826-6c19-43e0-8565-d357549c44ec None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:38.150861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:38.152754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:38.169161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c352839-73cd-48c6-840b-c0e07cf5fbc7 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:38.181870 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1aa826-6c19-43e0-8565-d357549c44ec None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:38.206356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665e9a93-8ccf-4dad-ad87-074ae32b98c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:38.207070 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665e9a93-8ccf-4dad-ad87-074ae32b98c7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:38.207346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665e9a93-8ccf-4dad-ad87-074ae32b98c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:38.207613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665e9a93-8ccf-4dad-ad87-074ae32b98c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:38.208253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665e9a93-8ccf-4dad-ad87-074ae32b98c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:38.208548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665e9a93-8ccf-4dad-ad87-074ae32b98c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:38.210123 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665e9a93-8ccf-4dad-ad87-074ae32b98c7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:38.217808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7386833-159c-489a-b778-f5fc3f3d10fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:38.218444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7386833-159c-489a-b778-f5fc3f3d10fe None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:38.218774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7386833-159c-489a-b778-f5fc3f3d10fe None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:38.219049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7386833-159c-489a-b778-f5fc3f3d10fe None admin] PATH_INFO: `/v3/endpoints` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:38.219755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7386833-159c-489a-b778-f5fc3f3d10fe None admin] RBAC: Adding query filter params (interface=public, service_id=b22bce89ec0044ff8ee7d75583819a91, region_id=RegionOne) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:38.220058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7386833-159c-489a-b778-f5fc3f3d10fe None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:38.222135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7386833-159c-489a-b778-f5fc3f3d10fe None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:38.224457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 458/940] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:38.230891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665e9a93-8ccf-4dad-ad87-074ae32b98c7 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:38.233596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 483/941] 10.222.0.65 () {56 vars in 1251 bytes} [Wed Aug 24 03:27:38 2022] GET /identity/v3/endpoints?service_id=b22bce89ec0044ff8ee7d75583819a91&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:38.239153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7386833-159c-489a-b778-f5fc3f3d10fe None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:38.344158 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665e9a93-8ccf-4dad-ad87-074ae32b98c7 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:38.349431 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7386833-159c-489a-b778-f5fc3f3d10fe None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:38.380356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa79b08-f004-4f44-853d-9b436f7450e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:38.381236 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa79b08-f004-4f44-853d-9b436f7450e6 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:38.381613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa79b08-f004-4f44-853d-9b436f7450e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:38.381902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa79b08-f004-4f44-853d-9b436f7450e6 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:38.385756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf5e966-bea7-4716-94da-4e26c5f26806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:38.386390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf5e966-bea7-4716-94da-4e26c5f26806 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:38.386671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf5e966-bea7-4716-94da-4e26c5f26806 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:38.386940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf5e966-bea7-4716-94da-4e26c5f26806 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:38.387544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf5e966-bea7-4716-94da-4e26c5f26806 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:38.388900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf5e966-bea7-4716-94da-4e26c5f26806 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:38.389508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa79b08-f004-4f44-853d-9b436f7450e6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:38.390961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa79b08-f004-4f44-853d-9b436f7450e6 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:38.391300 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:38.396115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 484/942] 10.222.0.65 () {56 vars in 1087 bytes} [Wed Aug 24 03:27:38 2022] GET /identity/v3/services => generated 1497 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:27:38.412554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108c7343-1828-4d58-82f7-7585efb019cb None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:38.412554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108c7343-1828-4d58-82f7-7585efb019cb None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:38.412554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108c7343-1828-4d58-82f7-7585efb019cb None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-afa79b08-f004-4f44-853d-9b436f7450e6 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:38.412983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:38.413866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:38.415282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:38.415282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:38.415282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:38.415282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:38.415282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:38.415282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:38.415282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:38.415282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:38.415282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 03:27:38.415282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:38.415282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 459/943] 10.222.0.65 () {56 vars in 1117 bytes} [Wed Aug 24 03:27:38 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:38.415947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 485/944] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:38 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:38.416506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22bd5f5-6ddb-43e1-b739-c2fe51c15228 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:38.416789 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22bd5f5-6ddb-43e1-b739-c2fe51c15228 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:38.417062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22bd5f5-6ddb-43e1-b739-c2fe51c15228 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:38.417175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108c7343-1828-4d58-82f7-7585efb019cb None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:38.425506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:38.441221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f22bd5f5-6ddb-43e1-b739-c2fe51c15228 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:38.442656 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f22bd5f5-6ddb-43e1-b739-c2fe51c15228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:38.538607 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 460/945] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:38 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:38.542816 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108c7343-1828-4d58-82f7-7585efb019cb None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:38.577494 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f1e34c-dd07-4ef5-86ac-55b7358c117a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:38.578099 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f1e34c-dd07-4ef5-86ac-55b7358c117a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:38.578406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f1e34c-dd07-4ef5-86ac-55b7358c117a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:38.578672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f1e34c-dd07-4ef5-86ac-55b7358c117a None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:38.579405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f1e34c-dd07-4ef5-86ac-55b7358c117a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:38.579703 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f1e34c-dd07-4ef5-86ac-55b7358c117a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:38.581845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f1e34c-dd07-4ef5-86ac-55b7358c117a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:38.599606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 486/946] 10.222.0.65 () {56 vars in 1149 bytes} [Wed Aug 24 03:27:38 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:38.605149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22bd5f5-6ddb-43e1-b739-c2fe51c15228 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:38.713287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22bd5f5-6ddb-43e1-b739-c2fe51c15228 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:38.753855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee77109-d85f-432c-bc59-9d14cecdedfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:38.754838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee77109-d85f-432c-bc59-9d14cecdedfa None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:38.755309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee77109-d85f-432c-bc59-9d14cecdedfa None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:38.756017 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee77109-d85f-432c-bc59-9d14cecdedfa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:38.758296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:38.768863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee77109-d85f-432c-bc59-9d14cecdedfa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:38.770620 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee77109-d85f-432c-bc59-9d14cecdedfa None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-fee77109-d85f-432c-bc59-9d14cecdedfa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:38.786696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:38.787687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:38.788583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:38.788583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:38.788583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:38.790370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 461/947] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:38.796537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f1e34c-dd07-4ef5-86ac-55b7358c117a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:38.848206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee77109-d85f-432c-bc59-9d14cecdedfa None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:38.860590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:38.905833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f1e34c-dd07-4ef5-86ac-55b7358c117a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:38.943249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede7b7fd-ec21-43b0-9df0-4584a2225ea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:38.944243 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede7b7fd-ec21-43b0-9df0-4584a2225ea6 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:38.944761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede7b7fd-ec21-43b0-9df0-4584a2225ea6 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:38.945257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede7b7fd-ec21-43b0-9df0-4584a2225ea6 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:38.946189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede7b7fd-ec21-43b0-9df0-4584a2225ea6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:38.946748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede7b7fd-ec21-43b0-9df0-4584a2225ea6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:38.948325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede7b7fd-ec21-43b0-9df0-4584a2225ea6 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:38.965360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 487/948] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:38.967029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede7b7fd-ec21-43b0-9df0-4584a2225ea6 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:38.969252 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee77109-d85f-432c-bc59-9d14cecdedfa None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:39.007928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b63121f-6e0e-4908-b33d-b1bea69491e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:39.008539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b63121f-6e0e-4908-b33d-b1bea69491e9 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:39.008817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b63121f-6e0e-4908-b33d-b1bea69491e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:39.009086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b63121f-6e0e-4908-b33d-b1bea69491e9 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:39.009734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b63121f-6e0e-4908-b33d-b1bea69491e9 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:39.011315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b63121f-6e0e-4908-b33d-b1bea69491e9 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:39.014036 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: WARNING keystone.server.flask.common [None req-1b63121f-6e0e-4908-b33d-b1bea69491e9 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 24 03:27:39.073758 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede7b7fd-ec21-43b0-9df0-4584a2225ea6 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:39.076550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.sql.core [None req-1b63121f-6e0e-4908-b33d-b1bea69491e9 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 24 03:27:39.076550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 24 03:27:39.076550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [parameters: {'id': '547e301419944b89aefbc18651291d6d', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 24 03:27:39.076550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81658) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 24 03:27:39.094962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-1b63121f-6e0e-4908-b33d-b1bea69491e9 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 547e301419944b89aefbc18651291d6d.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 547e301419944b89aefbc18651291d6d. Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application result = self._query(query) Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application conn.query(q) Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 24 03:27:39.100443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application result.read() Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Aug 24 03:27:39.101278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return project_ref.to_dict() Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application next(self.gen) Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application yield resource Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application next(self.gen) Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self.session.rollback() Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self.force_reraise() Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise self.value Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application session.commit() Aug 24 03:27:39.102288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._prepare_impl() Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self.session.flush() Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._flush(objects) Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application compat.raise_( Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application flush_context.execute() Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 24 03:27:39.103912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rec.execute(self) Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 24 03:27:39.104836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application result = self._query(query) Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application conn.query(q) Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application result.read() Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 24 03:27:39.105631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [parameters: {'id': '547e301419944b89aefbc18651291d6d', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:39.107518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:39.109670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:39.113307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:39.113307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Aug 24 03:27:39.113307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Aug 24 03:27:39.113307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:39.113307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:39.113307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Aug 24 03:27:39.113307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.Conflict( Aug 24 03:27:39.113307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 547e301419944b89aefbc18651291d6d. Aug 24 03:27:39.113307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:39.113307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 462/949] 10.222.0.65 () {60 vars in 1140 bytes} [Wed Aug 24 03:27:38 2022] POST /identity/v3/projects => generated 262 bytes in 259 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:39.113307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b63121f-6e0e-4908-b33d-b1bea69491e9 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:39.132110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f033185-f50f-4f15-aa67-ed54792cdee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:39.133221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f033185-f50f-4f15-aa67-ed54792cdee2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:39.133992 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f033185-f50f-4f15-aa67-ed54792cdee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:39.134580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f033185-f50f-4f15-aa67-ed54792cdee2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:39.144146 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f033185-f50f-4f15-aa67-ed54792cdee2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:39.146471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f033185-f50f-4f15-aa67-ed54792cdee2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-0f033185-f50f-4f15-aa67-ed54792cdee2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:39.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:39.162587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:39.163536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:39.163536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:39.163536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:39.163536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:39.163536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:39.163536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:39.163536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:39.163536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:39.166202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:39.170253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 488/950] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:39.172620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f033185-f50f-4f15-aa67-ed54792cdee2 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:39.243922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b63121f-6e0e-4908-b33d-b1bea69491e9 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:39.280729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4546f76-828f-4239-adb1-49e081b698a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:39.281741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4546f76-828f-4239-adb1-49e081b698a4 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:39.282291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4546f76-828f-4239-adb1-49e081b698a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:39.282963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4546f76-828f-4239-adb1-49e081b698a4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:39.291402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4546f76-828f-4239-adb1-49e081b698a4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:39.292721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f033185-f50f-4f15-aa67-ed54792cdee2 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:39.293640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4546f76-828f-4239-adb1-49e081b698a4 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-a4546f76-828f-4239-adb1-49e081b698a4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:39.305714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:39.307001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:39.307978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:39.307978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:39.307978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:39.307978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:39.307978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:39.307978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:39.307978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:39.307978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:39.310552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 463/951] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:39.316951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4546f76-828f-4239-adb1-49e081b698a4 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:39.334969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bf5dc7-f6b0-4c7c-911a-516b546e0a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:39.335978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bf5dc7-f6b0-4c7c-911a-516b546e0a35 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:39.335978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bf5dc7-f6b0-4c7c-911a-516b546e0a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:39.336234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bf5dc7-f6b0-4c7c-911a-516b546e0a35 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:39.336902 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bf5dc7-f6b0-4c7c-911a-516b546e0a35 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:39.337207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bf5dc7-f6b0-4c7c-911a-516b546e0a35 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:39.338933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bf5dc7-f6b0-4c7c-911a-516b546e0a35 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:39.353176 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 489/952] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:39.354453 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29fae04-a5ad-48ee-8c3a-7b13c47dc09f None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:39.354737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29fae04-a5ad-48ee-8c3a-7b13c47dc09f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:39.355011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29fae04-a5ad-48ee-8c3a-7b13c47dc09f None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:39.356673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 490/953] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:39.360648 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29fae04-a5ad-48ee-8c3a-7b13c47dc09f None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:39.405264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:39.429953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4546f76-828f-4239-adb1-49e081b698a4 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:39.431652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:39.465965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a520c2b7-e414-447c-bd9f-6764b661d37c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:39.466620 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a520c2b7-e414-447c-bd9f-6764b661d37c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:39.466897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a520c2b7-e414-447c-bd9f-6764b661d37c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:39.467167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a520c2b7-e414-447c-bd9f-6764b661d37c None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:39.467824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a520c2b7-e414-447c-bd9f-6764b661d37c None admin] RBAC: Adding query filter params (name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:39.468121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a520c2b7-e414-447c-bd9f-6764b661d37c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:39.469539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a520c2b7-e414-447c-bd9f-6764b661d37c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:39.476483 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29fae04-a5ad-48ee-8c3a-7b13c47dc09f None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:39.492317 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 464/954] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:39 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:39.494061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf9502-0f63-4bc0-8a94-b1fea9c7078d None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:39.494379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf9502-0f63-4bc0-8a94-b1fea9c7078d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:39.494662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf9502-0f63-4bc0-8a94-b1fea9c7078d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:39.517214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-20bf9502-0f63-4bc0-8a94-b1fea9c7078d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:39.519287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-20bf9502-0f63-4bc0-8a94-b1fea9c7078d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:39.526552 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3b6aee-f20f-4644-afa7-b4bf7669e591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:39.527612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3b6aee-f20f-4644-afa7-b4bf7669e591 None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:39.528128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3b6aee-f20f-4644-afa7-b4bf7669e591 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:39.529209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3b6aee-f20f-4644-afa7-b4bf7669e591 None admin] PATH_INFO: `/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/c63ade06fed946e79dc1e4b5ecfab5e6/roles/1489a00dd22146ecb6161ff7d549f95c` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:39.532160 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3b6aee-f20f-4644-afa7-b4bf7669e591 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c653d5fa4304f298a295ce635efd5e6, user_id=c63ade06fed946e79dc1e4b5ecfab5e6, role_id=1489a00dd22146ecb6161ff7d549f95c)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:39.535383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3b6aee-f20f-4644-afa7-b4bf7669e591 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:39.604624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 491/955] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:27:39 2022] PUT /identity/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/c63ade06fed946e79dc1e4b5ecfab5e6/roles/1489a00dd22146ecb6161ff7d549f95c => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:27:39.606300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 465/956] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:39 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:39.871437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3b6aee-f20f-4644-afa7-b4bf7669e591 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:39.979542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:39.984517 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3b6aee-f20f-4644-afa7-b4bf7669e591 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:40.020276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c299b725-31ef-4dd1-b3c4-cbcb02c4f431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:40.021276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c299b725-31ef-4dd1-b3c4-cbcb02c4f431 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:40.021761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c299b725-31ef-4dd1-b3c4-cbcb02c4f431 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:40.022248 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c299b725-31ef-4dd1-b3c4-cbcb02c4f431 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:40.023472 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c299b725-31ef-4dd1-b3c4-cbcb02c4f431 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:40.025618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c299b725-31ef-4dd1-b3c4-cbcb02c4f431 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:40.091499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 492/957] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:27:39 2022] POST /identity/v3/services => generated 242 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:40.426685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c11cfd-2bd3-4e31-ac9c-9fd93b67f877 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:40.427142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c11cfd-2bd3-4e31-ac9c-9fd93b67f877 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:40.427220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c11cfd-2bd3-4e31-ac9c-9fd93b67f877 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:40.428822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 466/958] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:40.434519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00b0748-3306-47c5-8059-9f0f8f8d4a70 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:40.434806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00b0748-3306-47c5-8059-9f0f8f8d4a70 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:40.435084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00b0748-3306-47c5-8059-9f0f8f8d4a70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:40.453641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d00b0748-3306-47c5-8059-9f0f8f8d4a70 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:40.454875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d00b0748-3306-47c5-8059-9f0f8f8d4a70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:40.533045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 493/959] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:40 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:40.811607 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c11cfd-2bd3-4e31-ac9c-9fd93b67f877 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:40.910303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:40.920511 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c11cfd-2bd3-4e31-ac9c-9fd93b67f877 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:40.955333 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058ba5a2-b8b6-4f56-9037-674f3f8088ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:40.956444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058ba5a2-b8b6-4f56-9037-674f3f8088ee None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:40.956916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058ba5a2-b8b6-4f56-9037-674f3f8088ee None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:40.957373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058ba5a2-b8b6-4f56-9037-674f3f8088ee None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:40.958168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058ba5a2-b8b6-4f56-9037-674f3f8088ee None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:40.959817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058ba5a2-b8b6-4f56-9037-674f3f8088ee None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-058ba5a2-b8b6-4f56-9037-674f3f8088ee None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:40.969647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:40.970772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:40.972351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:40.975098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 467/960] 10.222.0.65 () {56 vars in 1126 bytes} [Wed Aug 24 03:27:40 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:40.980828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00b0748-3306-47c5-8059-9f0f8f8d4a70 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:41.073852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:41.091524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00b0748-3306-47c5-8059-9f0f8f8d4a70 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:41.128284 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efa612a-98af-4144-8360-a6a71354d12f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:41.129307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efa612a-98af-4144-8360-a6a71354d12f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:41.129781 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efa612a-98af-4144-8360-a6a71354d12f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:41.130292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efa612a-98af-4144-8360-a6a71354d12f None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:41.131383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efa612a-98af-4144-8360-a6a71354d12f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:41.131894 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efa612a-98af-4144-8360-a6a71354d12f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:41.133457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efa612a-98af-4144-8360-a6a71354d12f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:41.139228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 494/961] 10.222.0.65 () {56 vars in 1122 bytes} [Wed Aug 24 03:27:40 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:41.144787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058ba5a2-b8b6-4f56-9037-674f3f8088ee None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:41.261906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058ba5a2-b8b6-4f56-9037-674f3f8088ee None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:41.297745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf242453-564f-421f-80e2-3e991d044d81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:41.298379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf242453-564f-421f-80e2-3e991d044d81 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:41.298653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf242453-564f-421f-80e2-3e991d044d81 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:41.298919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf242453-564f-421f-80e2-3e991d044d81 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:41.300074 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf242453-564f-421f-80e2-3e991d044d81 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:41.300374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf242453-564f-421f-80e2-3e991d044d81 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:41.301806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf242453-564f-421f-80e2-3e991d044d81 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:41.307214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 468/962] 10.222.0.65 () {56 vars in 1122 bytes} [Wed Aug 24 03:27:41 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:41.312838 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efa612a-98af-4144-8360-a6a71354d12f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:41.417248 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efa612a-98af-4144-8360-a6a71354d12f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:41.451627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec506fc-6b4c-43c3-a7cf-c8e8f345f095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:41.452226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec506fc-6b4c-43c3-a7cf-c8e8f345f095 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:41.452503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec506fc-6b4c-43c3-a7cf-c8e8f345f095 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:41.452768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec506fc-6b4c-43c3-a7cf-c8e8f345f095 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:41.453432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec506fc-6b4c-43c3-a7cf-c8e8f345f095 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:41.454820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec506fc-6b4c-43c3-a7cf-c8e8f345f095 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:41.528318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e35290-e208-46c0-bea2-fa7cd24c7706 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:41.528853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e35290-e208-46c0-bea2-fa7cd24c7706 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:41.528853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e35290-e208-46c0-bea2-fa7cd24c7706 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:41.529781 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 495/963] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:27:41 2022] POST /identity/v3/endpoints => generated 349 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:41.530198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 469/964] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:41.536580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c21c85-66e1-4a05-bafe-27f0c89506b5 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:41.536915 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c21c85-66e1-4a05-bafe-27f0c89506b5 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:41.537189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c21c85-66e1-4a05-bafe-27f0c89506b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:41.545223 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:41.558655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-05c21c85-66e1-4a05-bafe-27f0c89506b5 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:41.559889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-05c21c85-66e1-4a05-bafe-27f0c89506b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:41.645276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 496/965] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:41 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:41.686369 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a36011-7a5d-4b51-96d2-b85a2568ff99 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:41.686554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a36011-7a5d-4b51-96d2-b85a2568ff99 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:41.686819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a36011-7a5d-4b51-96d2-b85a2568ff99 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:41.688426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 470/966] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:41.694005 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0510e0e3-25e7-41e7-8959-c774257fd17e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:41.694321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0510e0e3-25e7-41e7-8959-c774257fd17e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:41.694592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0510e0e3-25e7-41e7-8959-c774257fd17e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:41.713485 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-0510e0e3-25e7-41e7-8959-c774257fd17e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:41.714718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0510e0e3-25e7-41e7-8959-c774257fd17e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:41.788062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 497/967] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:41 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:41.915824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a36011-7a5d-4b51-96d2-b85a2568ff99 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:41.938979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:42.021739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a36011-7a5d-4b51-96d2-b85a2568ff99 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:42.057019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0405fc14-ae2c-42f2-b744-d82b37fb207c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:42.057627 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0405fc14-ae2c-42f2-b744-d82b37fb207c None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:42.057900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0405fc14-ae2c-42f2-b744-d82b37fb207c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:42.058177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0405fc14-ae2c-42f2-b744-d82b37fb207c None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:42.058830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0405fc14-ae2c-42f2-b744-d82b37fb207c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:42.060238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0405fc14-ae2c-42f2-b744-d82b37fb207c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:42.073145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0510e0e3-25e7-41e7-8959-c774257fd17e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:42.105182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.sql.core [None req-0405fc14-ae2c-42f2-b744-d82b37fb207c None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 24 03:27:42.105182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 24 03:27:42.105182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [parameters: {'user_id': '412569962126419097ad1b9a8125fe14', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 24 03:27:42.105182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81658) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-0405fc14-ae2c-42f2-b744-d82b37fb207c None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application result = self._query(query) Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application conn.query(q) Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application result.read() Aug 24 03:27:42.113879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 24 03:27:42.114953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application next(self.gen) Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application yield resource Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application next(self.gen) Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self.session.rollback() Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self.force_reraise() Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise self.value Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application session.commit() Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 24 03:27:42.116115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._prepare_impl() Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self.session.flush() Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._flush(objects) Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application compat.raise_( Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application flush_context.execute() Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rec.execute(self) Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 24 03:27:42.117186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application result = connection._execute_20( Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 24 03:27:42.118289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application result = self._query(query) Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application conn.query(q) Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application result.read() Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 24 03:27:42.119275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [parameters: {'user_id': '412569962126419097ad1b9a8125fe14', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:42.120285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Aug 24 03:27:42.121218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Aug 24 03:27:42.127901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Aug 24 03:27:42.127901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Aug 24 03:27:42.127901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 24 03:27:42.127901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 24 03:27:42.127901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 24 03:27:42.127901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:42.127901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 471/968] 10.222.0.65 () {60 vars in 1131 bytes} [Wed Aug 24 03:27:41 2022] POST /identity/v3/users => generated 160 bytes in 200 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:42.127901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b865a55d-7b70-4328-80bb-55940f1b7bf3 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:42.127901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b865a55d-7b70-4328-80bb-55940f1b7bf3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:42.127901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b865a55d-7b70-4328-80bb-55940f1b7bf3 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:42.129398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 472/969] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:42 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:42.129398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b865a55d-7b70-4328-80bb-55940f1b7bf3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:42.137526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:42.179185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0510e0e3-25e7-41e7-8959-c774257fd17e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:42.213833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061ed5d7-60f1-4898-a044-679c01d20e78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:42.214425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061ed5d7-60f1-4898-a044-679c01d20e78 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:42.214704 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061ed5d7-60f1-4898-a044-679c01d20e78 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:42.214974 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061ed5d7-60f1-4898-a044-679c01d20e78 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:42.218868 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:42.226197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061ed5d7-60f1-4898-a044-679c01d20e78 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-061ed5d7-60f1-4898-a044-679c01d20e78 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:42.237402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:42.238358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 498/970] 10.222.0.65 () {56 vars in 1120 bytes} [Wed Aug 24 03:27:42 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:42.239215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b465fc97-f391-4b87-83ef-cc1b4dc4816c None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:42.239991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b465fc97-f391-4b87-83ef-cc1b4dc4816c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:42.239991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b465fc97-f391-4b87-83ef-cc1b4dc4816c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:42.245574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b865a55d-7b70-4328-80bb-55940f1b7bf3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:42.255553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-b465fc97-f391-4b87-83ef-cc1b4dc4816c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:42.257007 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b465fc97-f391-4b87-83ef-cc1b4dc4816c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:42.287628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635f6d50-0742-4abe-a363-166bf6242188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:42.288257 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635f6d50-0742-4abe-a363-166bf6242188 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:42.288526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635f6d50-0742-4abe-a363-166bf6242188 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:42.288792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635f6d50-0742-4abe-a363-166bf6242188 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:42.296791 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635f6d50-0742-4abe-a363-166bf6242188 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:42.298355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635f6d50-0742-4abe-a363-166bf6242188 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-635f6d50-0742-4abe-a363-166bf6242188 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:42.310820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:42.311945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:42.313106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:42.313106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:42.313106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:42.313106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:42.313106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:42.313106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:42.313106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:42.313106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:42.313106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 24 03:27:42.313106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:42.313106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 473/971] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:27:42 2022] GET /identity/v3/users/cinder => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:42.318172 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635f6d50-0742-4abe-a363-166bf6242188 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:42.334357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 499/972] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:42 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:42.335766 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b465fc97-f391-4b87-83ef-cc1b4dc4816c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:42.418879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:42.432782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635f6d50-0742-4abe-a363-166bf6242188 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:42.469350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b465fc97-f391-4b87-83ef-cc1b4dc4816c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:42.471387 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d599d4-68ea-4e82-a4b6-2929efc43c1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:42.472606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d599d4-68ea-4e82-a4b6-2929efc43c1b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:42.472944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d599d4-68ea-4e82-a4b6-2929efc43c1b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:42.473289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d599d4-68ea-4e82-a4b6-2929efc43c1b None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:42.474014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d599d4-68ea-4e82-a4b6-2929efc43c1b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:42.474420 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d599d4-68ea-4e82-a4b6-2929efc43c1b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:42.475848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d599d4-68ea-4e82-a4b6-2929efc43c1b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:42.489809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 474/973] 10.222.0.65 () {56 vars in 1115 bytes} [Wed Aug 24 03:27:42 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:42.490838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d599d4-68ea-4e82-a4b6-2929efc43c1b None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:42.495598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:42.511941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c28190-f51e-4a88-87ba-7a0be485609c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:42.512590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c28190-f51e-4a88-87ba-7a0be485609c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:42.512869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c28190-f51e-4a88-87ba-7a0be485609c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:42.513146 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c28190-f51e-4a88-87ba-7a0be485609c None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:42.513830 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c28190-f51e-4a88-87ba-7a0be485609c None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:42.514126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c28190-f51e-4a88-87ba-7a0be485609c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:42.515645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c28190-f51e-4a88-87ba-7a0be485609c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:42.566580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 500/974] 10.222.0.65 () {56 vars in 1101 bytes} [Wed Aug 24 03:27:42 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:42.615352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d599d4-68ea-4e82-a4b6-2929efc43c1b None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:42.648785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c28190-f51e-4a88-87ba-7a0be485609c None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:42.658052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bde624-cbbf-4cd1-97fe-65d4c3be831a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:42.659243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bde624-cbbf-4cd1-97fe-65d4c3be831a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:42.659243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bde624-cbbf-4cd1-97fe-65d4c3be831a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:42.659517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bde624-cbbf-4cd1-97fe-65d4c3be831a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:42.671863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bde624-cbbf-4cd1-97fe-65d4c3be831a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:42.673346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bde624-cbbf-4cd1-97fe-65d4c3be831a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-25bde624-cbbf-4cd1-97fe-65d4c3be831a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:42.691222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:42.692296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:42.693220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:42.693220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:42.693220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:42.693220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 475/975] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:42.700833 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bde624-cbbf-4cd1-97fe-65d4c3be831a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:42.771880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:42.782431 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c28190-f51e-4a88-87ba-7a0be485609c None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:42.798788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:42.824636 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e47a46-e37e-4045-acce-c9fad7d00418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:42.825276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e47a46-e37e-4045-acce-c9fad7d00418 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:42.825554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e47a46-e37e-4045-acce-c9fad7d00418 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:42.825824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e47a46-e37e-4045-acce-c9fad7d00418 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:42.826440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e47a46-e37e-4045-acce-c9fad7d00418 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:42.827753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bde624-cbbf-4cd1-97fe-65d4c3be831a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:42.827927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e47a46-e37e-4045-acce-c9fad7d00418 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-27e47a46-e37e-4045-acce-c9fad7d00418 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:42.838930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:42.839996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:42.840886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 501/976] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:42 2022] GET /identity/v3/services/compute => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:42.848973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e47a46-e37e-4045-acce-c9fad7d00418 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:42.871073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b411ea-df44-40bc-b330-edc4d5e99d16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:42.872106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b411ea-df44-40bc-b330-edc4d5e99d16 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:42.872393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b411ea-df44-40bc-b330-edc4d5e99d16 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:42.872668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b411ea-df44-40bc-b330-edc4d5e99d16 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:42.873463 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b411ea-df44-40bc-b330-edc4d5e99d16 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:42.873766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b411ea-df44-40bc-b330-edc4d5e99d16 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:42.875171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b411ea-df44-40bc-b330-edc4d5e99d16 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:42.887939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 476/977] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:42.894540 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b411ea-df44-40bc-b330-edc4d5e99d16 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:42.958892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e47a46-e37e-4045-acce-c9fad7d00418 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:42.994912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bd9fdb-2261-4191-bcdc-5275f55c9985 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:42.995525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bd9fdb-2261-4191-bcdc-5275f55c9985 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:42.995802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bd9fdb-2261-4191-bcdc-5275f55c9985 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:42.996072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bd9fdb-2261-4191-bcdc-5275f55c9985 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:42.996699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bd9fdb-2261-4191-bcdc-5275f55c9985 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:42.997000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bd9fdb-2261-4191-bcdc-5275f55c9985 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:42.998389 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bd9fdb-2261-4191-bcdc-5275f55c9985 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:43.002524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b411ea-df44-40bc-b330-edc4d5e99d16 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:43.008208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 502/978] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:42 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:43.009686 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bd9fdb-2261-4191-bcdc-5275f55c9985 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:43.026899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:43.044803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83775190-5feb-45ba-b902-26bb995d46ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:43.045532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83775190-5feb-45ba-b902-26bb995d46ce None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:43.045714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83775190-5feb-45ba-b902-26bb995d46ce None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:43.045985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83775190-5feb-45ba-b902-26bb995d46ce None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:43.047880 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:43.053837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83775190-5feb-45ba-b902-26bb995d46ce None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:43.055387 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83775190-5feb-45ba-b902-26bb995d46ce None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-83775190-5feb-45ba-b902-26bb995d46ce None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:43.068421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:43.069538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:43.070525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:43.070525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:43.070525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:43.070525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:43.070525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:43.070525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:43.070525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:43.070525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:43.070525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 24 03:27:43.070525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:43.070525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 477/979] 10.222.0.65 () {56 vars in 1117 bytes} [Wed Aug 24 03:27:42 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:43.075676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83775190-5feb-45ba-b902-26bb995d46ce None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:43.119367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bd9fdb-2261-4191-bcdc-5275f55c9985 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:43.153946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9c44ff-7b60-4f94-9136-a0ed59dbb539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:43.154637 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9c44ff-7b60-4f94-9136-a0ed59dbb539 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:43.154840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9c44ff-7b60-4f94-9136-a0ed59dbb539 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:43.155099 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9c44ff-7b60-4f94-9136-a0ed59dbb539 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:43.155723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9c44ff-7b60-4f94-9136-a0ed59dbb539 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:43.156017 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9c44ff-7b60-4f94-9136-a0ed59dbb539 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:43.157405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9c44ff-7b60-4f94-9136-a0ed59dbb539 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:43.168059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 503/980] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:43 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:43.169035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9c44ff-7b60-4f94-9136-a0ed59dbb539 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:43.191754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83775190-5feb-45ba-b902-26bb995d46ce None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:43.231858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0013c13f-6bee-4d21-ac62-798fdae19879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:43.232473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0013c13f-6bee-4d21-ac62-798fdae19879 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:43.232801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0013c13f-6bee-4d21-ac62-798fdae19879 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:43.233011 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0013c13f-6bee-4d21-ac62-798fdae19879 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:43.233775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0013c13f-6bee-4d21-ac62-798fdae19879 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:43.234077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0013c13f-6bee-4d21-ac62-798fdae19879 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:43.235566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0013c13f-6bee-4d21-ac62-798fdae19879 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:43.261460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 478/981] 10.222.0.65 () {56 vars in 1149 bytes} [Wed Aug 24 03:27:43 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:43.262235 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0013c13f-6bee-4d21-ac62-798fdae19879 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:43.284907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:43.288797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9c44ff-7b60-4f94-9136-a0ed59dbb539 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:43.300753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:43.324720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bed00e-4591-4996-b302-1b89a0dfdf1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:43.325330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bed00e-4591-4996-b302-1b89a0dfdf1b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:43.325608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bed00e-4591-4996-b302-1b89a0dfdf1b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:43.325878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bed00e-4591-4996-b302-1b89a0dfdf1b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:43.326664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bed00e-4591-4996-b302-1b89a0dfdf1b None admin] RBAC: Adding query filter params (interface=public, service_id=1cc6a411b2ed445f95ca4ce65ede10c6, region_id=RegionOne) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:43.326972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bed00e-4591-4996-b302-1b89a0dfdf1b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:43.328330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62bed00e-4591-4996-b302-1b89a0dfdf1b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:43.334027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 504/982] 10.222.0.65 () {56 vars in 1251 bytes} [Wed Aug 24 03:27:43 2022] GET /identity/v3/endpoints?service_id=1cc6a411b2ed445f95ca4ce65ede10c6&interface=public®ion_id=RegionOne => generated 194 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:43.339622 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bed00e-4591-4996-b302-1b89a0dfdf1b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:43.372286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0013c13f-6bee-4d21-ac62-798fdae19879 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:43.415918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f8f2e5-fa0f-495d-9a5b-f4a2805aad6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:43.416537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f8f2e5-fa0f-495d-9a5b-f4a2805aad6a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:43.416814 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f8f2e5-fa0f-495d-9a5b-f4a2805aad6a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:43.417080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f8f2e5-fa0f-495d-9a5b-f4a2805aad6a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:43.448031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62bed00e-4591-4996-b302-1b89a0dfdf1b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:43.484327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9619cf2-6f21-40ed-978f-d1a25dfb8917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:43.484939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9619cf2-6f21-40ed-978f-d1a25dfb8917 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:43.485221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9619cf2-6f21-40ed-978f-d1a25dfb8917 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:43.485492 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9619cf2-6f21-40ed-978f-d1a25dfb8917 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:43.486095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9619cf2-6f21-40ed-978f-d1a25dfb8917 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:43.487189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f8f2e5-fa0f-495d-9a5b-f4a2805aad6a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:43.488028 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9619cf2-6f21-40ed-978f-d1a25dfb8917 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:43.488730 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f8f2e5-fa0f-495d-9a5b-f4a2805aad6a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:43.494850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 505/983] 10.222.0.65 () {56 vars in 1087 bytes} [Wed Aug 24 03:27:43 2022] GET /identity/v3/services => generated 1727 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:27:43.526833 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:43.545342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2665e61c-1a6a-4199-9c21-a6efe10db09d None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:43.545342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2665e61c-1a6a-4199-9c21-a6efe10db09d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:43.545342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2665e61c-1a6a-4199-9c21-a6efe10db09d None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:43.546219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 506/984] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:43.552904 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0407310c-a236-4d26-8a85-49ab2c386fce None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:43.553193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0407310c-a236-4d26-8a85-49ab2c386fce None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:43.553466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0407310c-a236-4d26-8a85-49ab2c386fce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:43.573061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-0407310c-a236-4d26-8a85-49ab2c386fce None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:43.574463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0407310c-a236-4d26-8a85-49ab2c386fce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:43.602233 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:43.611071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 479/985] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:43 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 24 03:27:43.617404 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f8f2e5-fa0f-495d-9a5b-f4a2805aad6a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:43.657768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 507/986] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:43 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:43.726994 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f8f2e5-fa0f-495d-9a5b-f4a2805aad6a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:43.761153 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f78938-26ee-45ef-8895-1f4c3ecf7667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:43.761785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f78938-26ee-45ef-8895-1f4c3ecf7667 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:43.762061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f78938-26ee-45ef-8895-1f4c3ecf7667 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:43.762361 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f78938-26ee-45ef-8895-1f4c3ecf7667 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:43.764208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:43.774293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f78938-26ee-45ef-8895-1f4c3ecf7667 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:43.775739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f78938-26ee-45ef-8895-1f4c3ecf7667 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-66f78938-26ee-45ef-8895-1f4c3ecf7667 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:43.790415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:43.791382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:43.792270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:43.792270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:43.792270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:43.792270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 480/987] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:43.797238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0407310c-a236-4d26-8a85-49ab2c386fce None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:43.900457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0407310c-a236-4d26-8a85-49ab2c386fce None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:43.925764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f78938-26ee-45ef-8895-1f4c3ecf7667 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:43.936680 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17ca8e7-d9e3-44f2-bd49-de5ddc48aedb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:43.937297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17ca8e7-d9e3-44f2-bd49-de5ddc48aedb None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:43.937574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17ca8e7-d9e3-44f2-bd49-de5ddc48aedb None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:43.937843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17ca8e7-d9e3-44f2-bd49-de5ddc48aedb None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:43.938587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17ca8e7-d9e3-44f2-bd49-de5ddc48aedb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:43.938890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17ca8e7-d9e3-44f2-bd49-de5ddc48aedb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:43.940800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17ca8e7-d9e3-44f2-bd49-de5ddc48aedb None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:43.952522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 508/988] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:43.958573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17ca8e7-d9e3-44f2-bd49-de5ddc48aedb None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:43.977978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:44.033759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f78938-26ee-45ef-8895-1f4c3ecf7667 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:44.065069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17ca8e7-d9e3-44f2-bd49-de5ddc48aedb None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:44.068633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21182926-5779-4c75-9d33-ff5768553c33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:44.069524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21182926-5779-4c75-9d33-ff5768553c33 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:44.069981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21182926-5779-4c75-9d33-ff5768553c33 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:44.070563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21182926-5779-4c75-9d33-ff5768553c33 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:44.071319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21182926-5779-4c75-9d33-ff5768553c33 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:44.072846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21182926-5779-4c75-9d33-ff5768553c33 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-21182926-5779-4c75-9d33-ff5768553c33 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:44.082448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:44.083435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:44.084286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:44.086193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 481/989] 10.222.0.65 () {56 vars in 1126 bytes} [Wed Aug 24 03:27:43 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:44.091964 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21182926-5779-4c75-9d33-ff5768553c33 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:44.100837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7a629b-b61e-4168-bf20-66bcc06e141f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:44.102086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7a629b-b61e-4168-bf20-66bcc06e141f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:44.102463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7a629b-b61e-4168-bf20-66bcc06e141f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:44.102808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7a629b-b61e-4168-bf20-66bcc06e141f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:44.113028 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7a629b-b61e-4168-bf20-66bcc06e141f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:44.114624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7a629b-b61e-4168-bf20-66bcc06e141f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-cd7a629b-b61e-4168-bf20-66bcc06e141f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:44.126741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:44.128309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:44.131237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:44.131237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:44.131237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:44.131237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:44.131237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:44.131237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:44.131237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:44.131237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:44.131237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 509/990] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:44.134565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7a629b-b61e-4168-bf20-66bcc06e141f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:44.146120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:44.201564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21182926-5779-4c75-9d33-ff5768553c33 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:44.217555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:44.237752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb05fb7-f9bc-43f3-9c20-03c51cb5ec43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:44.238384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb05fb7-f9bc-43f3-9c20-03c51cb5ec43 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:44.238658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb05fb7-f9bc-43f3-9c20-03c51cb5ec43 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:44.238924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb05fb7-f9bc-43f3-9c20-03c51cb5ec43 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:44.239030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7a629b-b61e-4168-bf20-66bcc06e141f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:44.239545 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb05fb7-f9bc-43f3-9c20-03c51cb5ec43 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:44.239841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb05fb7-f9bc-43f3-9c20-03c51cb5ec43 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:44.241202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb05fb7-f9bc-43f3-9c20-03c51cb5ec43 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:44.250552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 482/991] 10.222.0.65 () {56 vars in 1122 bytes} [Wed Aug 24 03:27:44 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:44.251566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb05fb7-f9bc-43f3-9c20-03c51cb5ec43 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:44.281024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c409d6-31fd-4dee-8b9a-8ee347dde789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:44.281627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c409d6-31fd-4dee-8b9a-8ee347dde789 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:44.281907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c409d6-31fd-4dee-8b9a-8ee347dde789 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:44.282198 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c409d6-31fd-4dee-8b9a-8ee347dde789 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:44.282886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c409d6-31fd-4dee-8b9a-8ee347dde789 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:44.283186 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c409d6-31fd-4dee-8b9a-8ee347dde789 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:44.284834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c409d6-31fd-4dee-8b9a-8ee347dde789 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:44.297889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 510/992] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:44.304038 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c409d6-31fd-4dee-8b9a-8ee347dde789 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:44.357060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb05fb7-f9bc-43f3-9c20-03c51cb5ec43 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:44.372114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:44.393768 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20537ef-11d6-46f6-a7fa-7c2c395d6aa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:44.394402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20537ef-11d6-46f6-a7fa-7c2c395d6aa3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:44.394684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20537ef-11d6-46f6-a7fa-7c2c395d6aa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:44.395006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20537ef-11d6-46f6-a7fa-7c2c395d6aa3 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:44.395637 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20537ef-11d6-46f6-a7fa-7c2c395d6aa3 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:44.395945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20537ef-11d6-46f6-a7fa-7c2c395d6aa3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:44.397318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20537ef-11d6-46f6-a7fa-7c2c395d6aa3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:44.406253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 483/993] 10.222.0.65 () {56 vars in 1122 bytes} [Wed Aug 24 03:27:44 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:44.412724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20537ef-11d6-46f6-a7fa-7c2c395d6aa3 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:44.417992 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c409d6-31fd-4dee-8b9a-8ee347dde789 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:44.454652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41774430-01d7-46e6-a528-7b520a4c2e23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:44.455827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41774430-01d7-46e6-a528-7b520a4c2e23 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:44.456082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41774430-01d7-46e6-a528-7b520a4c2e23 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:44.456452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41774430-01d7-46e6-a528-7b520a4c2e23 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:44.457630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41774430-01d7-46e6-a528-7b520a4c2e23 None admin] RBAC: Adding query filter params (role.id=1489a00dd22146ecb6161ff7d549f95c, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=c63ade06fed946e79dc1e4b5ecfab5e6) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:44.457942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41774430-01d7-46e6-a528-7b520a4c2e23 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:44.459809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41774430-01d7-46e6-a528-7b520a4c2e23 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:44.470855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 511/994] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:44 2022] GET /identity/v3/role_assignments?user.id=c63ade06fed946e79dc1e4b5ecfab5e6&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=1489a00dd22146ecb6161ff7d549f95c => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:44.526746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20537ef-11d6-46f6-a7fa-7c2c395d6aa3 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:44.564490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06f7b19-e95e-4f78-be33-082d49082c5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:44.565407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06f7b19-e95e-4f78-be33-082d49082c5c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:44.565885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06f7b19-e95e-4f78-be33-082d49082c5c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:44.566186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06f7b19-e95e-4f78-be33-082d49082c5c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:44.566919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06f7b19-e95e-4f78-be33-082d49082c5c None admin] RBAC: Adding query filter params (interface=admin, service_id=b22bce89ec0044ff8ee7d75583819a91, region_id=RegionOne) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:44.567227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06f7b19-e95e-4f78-be33-082d49082c5c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:44.568615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06f7b19-e95e-4f78-be33-082d49082c5c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:44.578397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 484/995] 10.222.0.65 () {56 vars in 1249 bytes} [Wed Aug 24 03:27:44 2022] GET /identity/v3/endpoints?service_id=b22bce89ec0044ff8ee7d75583819a91&interface=admin®ion_id=RegionOne => generated 193 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:44.580741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41774430-01d7-46e6-a528-7b520a4c2e23 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:44.583994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:44.690092 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41774430-01d7-46e6-a528-7b520a4c2e23 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:44.723354 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75303784-9df5-46a2-ac70-968ecbdaa7b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:44.723964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75303784-9df5-46a2-ac70-968ecbdaa7b2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:44.724245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75303784-9df5-46a2-ac70-968ecbdaa7b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:44.724519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75303784-9df5-46a2-ac70-968ecbdaa7b2 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:44.725125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75303784-9df5-46a2-ac70-968ecbdaa7b2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:44.726585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75303784-9df5-46a2-ac70-968ecbdaa7b2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:44.732886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 512/996] 10.222.0.65 () {56 vars in 1087 bytes} [Wed Aug 24 03:27:44 2022] GET /identity/v3/services => generated 1727 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:27:45.434545 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d368cb66-b176-4fff-96c7-7dce85b9459a None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:45.434974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d368cb66-b176-4fff-96c7-7dce85b9459a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:45.435151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d368cb66-b176-4fff-96c7-7dce85b9459a None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:45.436564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 485/997] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:45.442605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fcf1a-9ed9-49d5-af93-4bd8719c0c69 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:45.442998 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fcf1a-9ed9-49d5-af93-4bd8719c0c69 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:45.443173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fcf1a-9ed9-49d5-af93-4bd8719c0c69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:45.461078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-193fcf1a-9ed9-49d5-af93-4bd8719c0c69 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:45.462312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-193fcf1a-9ed9-49d5-af93-4bd8719c0c69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:45.525899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:45.534526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 513/998] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:45 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:45.801156 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d368cb66-b176-4fff-96c7-7dce85b9459a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:45.849324 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:45.907762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d368cb66-b176-4fff-96c7-7dce85b9459a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:45.941871 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1844ae8a-6a09-4622-b544-8f8b53139afa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:45.942523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844ae8a-6a09-4622-b544-8f8b53139afa None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:45.942800 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844ae8a-6a09-4622-b544-8f8b53139afa None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:45.943069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844ae8a-6a09-4622-b544-8f8b53139afa None admin] PATH_INFO: `/v3/services/compute` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:45.943645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1844ae8a-6a09-4622-b544-8f8b53139afa None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:45.944987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1844ae8a-6a09-4622-b544-8f8b53139afa None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-1844ae8a-6a09-4622-b544-8f8b53139afa None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:45.953521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:45.954561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 24 03:27:45.955655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:45.958471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 486/999] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:45 2022] GET /identity/v3/services/compute => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:45.960762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193fcf1a-9ed9-49d5-af93-4bd8719c0c69 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:46.061618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193fcf1a-9ed9-49d5-af93-4bd8719c0c69 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:46.095430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b52d35f-cb90-4d5b-aa01-8e6b844fd3a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:46.096570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b52d35f-cb90-4d5b-aa01-8e6b844fd3a3 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:46.096850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b52d35f-cb90-4d5b-aa01-8e6b844fd3a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:46.097121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b52d35f-cb90-4d5b-aa01-8e6b844fd3a3 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:46.097758 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b52d35f-cb90-4d5b-aa01-8e6b844fd3a3 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:46.098060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b52d35f-cb90-4d5b-aa01-8e6b844fd3a3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:46.099467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b52d35f-cb90-4d5b-aa01-8e6b844fd3a3 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:46.108924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 514/1000] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:45 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:46.110473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1844ae8a-6a09-4622-b544-8f8b53139afa None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:46.214035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1844ae8a-6a09-4622-b544-8f8b53139afa None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:46.222617 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:46.249431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c8fb62-7ba7-4cfc-a60e-641d257b512b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:46.250131 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c8fb62-7ba7-4cfc-a60e-641d257b512b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:46.250629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c8fb62-7ba7-4cfc-a60e-641d257b512b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:46.250964 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c8fb62-7ba7-4cfc-a60e-641d257b512b None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:46.251761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c8fb62-7ba7-4cfc-a60e-641d257b512b None admin] RBAC: Adding query filter params (type=compute) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:46.252711 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c8fb62-7ba7-4cfc-a60e-641d257b512b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:46.254378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c8fb62-7ba7-4cfc-a60e-641d257b512b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:46.264447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 487/1001] 10.222.0.65 () {56 vars in 1112 bytes} [Wed Aug 24 03:27:46 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:46.265736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b52d35f-cb90-4d5b-aa01-8e6b844fd3a3 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:46.328318 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:46.370370 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b52d35f-cb90-4d5b-aa01-8e6b844fd3a3 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:46.410411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cda585-303d-4e47-907e-b2a41915efdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:46.411040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cda585-303d-4e47-907e-b2a41915efdb None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:46.411324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cda585-303d-4e47-907e-b2a41915efdb None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:46.411601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cda585-303d-4e47-907e-b2a41915efdb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:46.412296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cda585-303d-4e47-907e-b2a41915efdb None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:46.413669 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cda585-303d-4e47-907e-b2a41915efdb None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:46.415647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1066c704-3ea9-441d-a43c-d2e00a6cbbbb None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:46.415923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1066c704-3ea9-441d-a43c-d2e00a6cbbbb None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:46.416188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1066c704-3ea9-441d-a43c-d2e00a6cbbbb None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:46.417515 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 488/1002] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:46.422722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c807d43-58e3-42af-b7e0-2e7216e9069c None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:46.423064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c807d43-58e3-42af-b7e0-2e7216e9069c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:46.423402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c807d43-58e3-42af-b7e0-2e7216e9069c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:46.443934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-8c807d43-58e3-42af-b7e0-2e7216e9069c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:46.445708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-8c807d43-58e3-42af-b7e0-2e7216e9069c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:46.471819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 515/1003] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:27:46 2022] POST /identity/v3/endpoints => generated 334 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:46.534972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 489/1004] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:46 2022] POST /identity/v3/auth/tokens => generated 1726 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:46.669852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9727e252-2cb3-4389-8323-fdd07276fb5a None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:46.670141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9727e252-2cb3-4389-8323-fdd07276fb5a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:46.670448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9727e252-2cb3-4389-8323-fdd07276fb5a None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:46.671785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 516/1005] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:46.677534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0f8c15-0c4c-4580-a0f6-a5a58157f8f6 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:46.677812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0f8c15-0c4c-4580-a0f6-a5a58157f8f6 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:46.678081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0f8c15-0c4c-4580-a0f6-a5a58157f8f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:46.696386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-8b0f8c15-0c4c-4580-a0f6-a5a58157f8f6 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:46.697788 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-8b0f8c15-0c4c-4580-a0f6-a5a58157f8f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:46.767172 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 490/1006] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:46 2022] POST /identity/v3/auth/tokens => generated 1726 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:46.802130 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9727e252-2cb3-4389-8323-fdd07276fb5a None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:46.903737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9727e252-2cb3-4389-8323-fdd07276fb5a None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:46.924056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:46.942715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310deb81-b43a-4a12-973b-7ee4fe47ba32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:46.945225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310deb81-b43a-4a12-973b-7ee4fe47ba32 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:46.945225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310deb81-b43a-4a12-973b-7ee4fe47ba32 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:46.945225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310deb81-b43a-4a12-973b-7ee4fe47ba32 None admin] PATH_INFO: `/v3/services/image` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:46.945225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310deb81-b43a-4a12-973b-7ee4fe47ba32 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:46.945945 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310deb81-b43a-4a12-973b-7ee4fe47ba32 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-310deb81-b43a-4a12-973b-7ee4fe47ba32 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:46.954606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:46.956513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:46.957447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 517/1007] 10.222.0.65 () {56 vars in 1105 bytes} [Wed Aug 24 03:27:46 2022] GET /identity/v3/services/image => generated 86 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:46.962258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0f8c15-0c4c-4580-a0f6-a5a58157f8f6 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:46.989353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:47.036747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310deb81-b43a-4a12-973b-7ee4fe47ba32 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:47.069654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0f8c15-0c4c-4580-a0f6-a5a58157f8f6 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:47.104411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675a6879-efbf-44f6-94fb-e4681399ddb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:47.105081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675a6879-efbf-44f6-94fb-e4681399ddb6 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:47.105357 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675a6879-efbf-44f6-94fb-e4681399ddb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:47.105629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675a6879-efbf-44f6-94fb-e4681399ddb6 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:47.106341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675a6879-efbf-44f6-94fb-e4681399ddb6 None admin] RBAC: Adding query filter params (name=image) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:47.106643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675a6879-efbf-44f6-94fb-e4681399ddb6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:47.108013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675a6879-efbf-44f6-94fb-e4681399ddb6 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:47.114305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 491/1008] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:46 2022] GET /identity/v3/services?name=image => generated 122 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:47.120038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675a6879-efbf-44f6-94fb-e4681399ddb6 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:47.152183 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310deb81-b43a-4a12-973b-7ee4fe47ba32 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:47.189550 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8c67a6-f112-4b29-80c8-a145e9fb98fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:47.191132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8c67a6-f112-4b29-80c8-a145e9fb98fe None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:47.191647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8c67a6-f112-4b29-80c8-a145e9fb98fe None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:47.192118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8c67a6-f112-4b29-80c8-a145e9fb98fe None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:47.192916 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8c67a6-f112-4b29-80c8-a145e9fb98fe None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:47.194399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8c67a6-f112-4b29-80c8-a145e9fb98fe None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-5f8c67a6-f112-4b29-80c8-a145e9fb98fe None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:47.203319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:47.204332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:47.205412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 518/1009] 10.222.0.65 () {56 vars in 1126 bytes} [Wed Aug 24 03:27:47 2022] GET /identity/v3/services/object-store => generated 93 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:47.210609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8c67a6-f112-4b29-80c8-a145e9fb98fe None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:47.217162 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:47.231237 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675a6879-efbf-44f6-94fb-e4681399ddb6 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:47.259461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:47.272397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaf4ca4-e1e5-49eb-a8f4-faa95a2a98c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:47.273222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaf4ca4-e1e5-49eb-a8f4-faa95a2a98c0 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:47.273331 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaf4ca4-e1e5-49eb-a8f4-faa95a2a98c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:47.273527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaf4ca4-e1e5-49eb-a8f4-faa95a2a98c0 None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:47.274194 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaf4ca4-e1e5-49eb-a8f4-faa95a2a98c0 None admin] RBAC: Adding query filter params (type=image) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:47.274523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaf4ca4-e1e5-49eb-a8f4-faa95a2a98c0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:47.275874 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaf4ca4-e1e5-49eb-a8f4-faa95a2a98c0 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:47.286312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 492/1010] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:47 2022] GET /identity/v3/services?type=image => generated 122 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:47.322706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8c67a6-f112-4b29-80c8-a145e9fb98fe None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:47.359993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6186eabc-e73f-4c90-98de-015711694ca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:47.360614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6186eabc-e73f-4c90-98de-015711694ca2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:47.360895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6186eabc-e73f-4c90-98de-015711694ca2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:47.361169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6186eabc-e73f-4c90-98de-015711694ca2 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:47.362740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6186eabc-e73f-4c90-98de-015711694ca2 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:47.363069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6186eabc-e73f-4c90-98de-015711694ca2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:47.364467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6186eabc-e73f-4c90-98de-015711694ca2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:47.370056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 519/1011] 10.222.0.65 () {56 vars in 1122 bytes} [Wed Aug 24 03:27:47 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:47.375811 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaf4ca4-e1e5-49eb-a8f4-faa95a2a98c0 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:47.492434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaf4ca4-e1e5-49eb-a8f4-faa95a2a98c0 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:47.528147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29400a3c-0b58-4851-930d-df2021b477be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:47.528752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29400a3c-0b58-4851-930d-df2021b477be None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:47.529027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29400a3c-0b58-4851-930d-df2021b477be None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:47.529293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29400a3c-0b58-4851-930d-df2021b477be None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:47.529921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29400a3c-0b58-4851-930d-df2021b477be None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:47.530246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29400a3c-0b58-4851-930d-df2021b477be None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:47.531601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29400a3c-0b58-4851-930d-df2021b477be None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:47.541849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 493/1012] 10.222.0.65 () {56 vars in 1122 bytes} [Wed Aug 24 03:27:47 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:47.543076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6186eabc-e73f-4c90-98de-015711694ca2 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:47.648826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6186eabc-e73f-4c90-98de-015711694ca2 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:47.657736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:47.682187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d330610-6ceb-47b1-933c-498396b14988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:47.682798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d330610-6ceb-47b1-933c-498396b14988 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:47.694828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d330610-6ceb-47b1-933c-498396b14988 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:47.694828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d330610-6ceb-47b1-933c-498396b14988 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:47.694828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d330610-6ceb-47b1-933c-498396b14988 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:47.694828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d330610-6ceb-47b1-933c-498396b14988 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:47.780082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 520/1013] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:27:47 2022] POST /identity/v3/endpoints => generated 325 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:48.528753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6044d8d6-cb30-4efc-ab8b-2d024318af63 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:48.529313 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6044d8d6-cb30-4efc-ab8b-2d024318af63 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:48.529313 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6044d8d6-cb30-4efc-ab8b-2d024318af63 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:48.530698 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 494/1014] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:48.536450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237e058f-7efa-491d-920b-d08914e104f1 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:48.536744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237e058f-7efa-491d-920b-d08914e104f1 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:48.537020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237e058f-7efa-491d-920b-d08914e104f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:48.555373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-237e058f-7efa-491d-920b-d08914e104f1 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:48.556564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-237e058f-7efa-491d-920b-d08914e104f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:48.629791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 521/1015] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:48 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:48.897912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6044d8d6-cb30-4efc-ab8b-2d024318af63 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:48.918474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:49.004549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6044d8d6-cb30-4efc-ab8b-2d024318af63 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:49.039121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9c5e86-d6a0-4bb6-9c91-26f7521b16df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:49.039745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9c5e86-d6a0-4bb6-9c91-26f7521b16df None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:49.040019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9c5e86-d6a0-4bb6-9c91-26f7521b16df None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:49.040348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9c5e86-d6a0-4bb6-9c91-26f7521b16df None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:49.050749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9c5e86-d6a0-4bb6-9c91-26f7521b16df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-2d9c5e86-d6a0-4bb6-9c91-26f7521b16df None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:49.055477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:49.056418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:49.057276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:49.057276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:49.057276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:49.057276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:49.057276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:49.057276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:49.057276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:49.057276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:49.057276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:49.057276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:49.057276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 03:27:49.057276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:49.058713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 495/1016] 10.222.0.65 () {56 vars in 1120 bytes} [Wed Aug 24 03:27:48 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:49.062526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237e058f-7efa-491d-920b-d08914e104f1 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:49.158492 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:49.164950 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237e058f-7efa-491d-920b-d08914e104f1 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:49.197902 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651420f3-1000-42c6-abe3-b3c5dad3922a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:49.198537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651420f3-1000-42c6-abe3-b3c5dad3922a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:49.198815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651420f3-1000-42c6-abe3-b3c5dad3922a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:49.199086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651420f3-1000-42c6-abe3-b3c5dad3922a None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:49.199748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651420f3-1000-42c6-abe3-b3c5dad3922a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:49.200047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651420f3-1000-42c6-abe3-b3c5dad3922a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:49.201447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651420f3-1000-42c6-abe3-b3c5dad3922a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:49.214847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 522/1017] 10.222.0.65 () {56 vars in 1115 bytes} [Wed Aug 24 03:27:49 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:49.216096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9c5e86-d6a0-4bb6-9c91-26f7521b16df None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:49.216706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64997547-4147-450b-bbda-ac90e492a4f0 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:49.216991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64997547-4147-450b-bbda-ac90e492a4f0 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:49.217263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64997547-4147-450b-bbda-ac90e492a4f0 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:49.218787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 523/1018] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:49.224355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e077bb-8a68-4ce3-a570-db20717b24d8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:49.224641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e077bb-8a68-4ce3-a570-db20717b24d8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:49.225258 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e077bb-8a68-4ce3-a570-db20717b24d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:49.244892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-57e077bb-8a68-4ce3-a570-db20717b24d8 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:49.246113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-57e077bb-8a68-4ce3-a570-db20717b24d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:49.313777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:49.319602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 524/1019] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:49 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:49.338340 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9c5e86-d6a0-4bb6-9c91-26f7521b16df None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:49.374206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75f6d9e-a70e-4d24-822b-72f1dba9e0e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:49.374833 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f6d9e-a70e-4d24-822b-72f1dba9e0e7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:49.375110 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f6d9e-a70e-4d24-822b-72f1dba9e0e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:49.375380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f6d9e-a70e-4d24-822b-72f1dba9e0e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:49.386359 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75f6d9e-a70e-4d24-822b-72f1dba9e0e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:49.387793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75f6d9e-a70e-4d24-822b-72f1dba9e0e7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-d75f6d9e-a70e-4d24-822b-72f1dba9e0e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:49.402491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:49.403505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:49.404410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:49.404410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:49.404410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:49.409974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 496/1020] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:49.410115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e077bb-8a68-4ce3-a570-db20717b24d8 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:49.496948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:49.511720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e077bb-8a68-4ce3-a570-db20717b24d8 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:49.546793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8c029a-2ca2-4bb9-a509-0ec5f96c7cbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:49.547401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8c029a-2ca2-4bb9-a509-0ec5f96c7cbe None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:49.547681 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8c029a-2ca2-4bb9-a509-0ec5f96c7cbe None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:49.547952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8c029a-2ca2-4bb9-a509-0ec5f96c7cbe None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:49.548619 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8c029a-2ca2-4bb9-a509-0ec5f96c7cbe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:49.549449 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8c029a-2ca2-4bb9-a509-0ec5f96c7cbe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:49.550943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8c029a-2ca2-4bb9-a509-0ec5f96c7cbe None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:49.562191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 525/1021] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:49.568258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75f6d9e-a70e-4d24-822b-72f1dba9e0e7 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:49.590981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8c029a-2ca2-4bb9-a509-0ec5f96c7cbe None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:49.674403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75f6d9e-a70e-4d24-822b-72f1dba9e0e7 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:49.695706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8c029a-2ca2-4bb9-a509-0ec5f96c7cbe None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:49.703043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:49.710542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae428648-25b1-4f90-b689-c5a6d0e4d585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:49.711144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae428648-25b1-4f90-b689-c5a6d0e4d585 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:49.711427 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae428648-25b1-4f90-b689-c5a6d0e4d585 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:49.711695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae428648-25b1-4f90-b689-c5a6d0e4d585 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:49.719445 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae428648-25b1-4f90-b689-c5a6d0e4d585 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:49.720916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae428648-25b1-4f90-b689-c5a6d0e4d585 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-ae428648-25b1-4f90-b689-c5a6d0e4d585 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:49.732604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:49.733602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:49.735168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:49.735168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:49.735168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:49.735168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:49.735168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:49.735168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:49.735168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:49.735168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:49.735168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:49.735168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:49.735847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e30c543-cf06-4694-8768-441e021bb645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:49.735847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e30c543-cf06-4694-8768-441e021bb645 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:49.736026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 497/1022] 10.222.0.65 () {56 vars in 1093 bytes} [Wed Aug 24 03:27:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:49.736026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2306e8c5-cb64-4644-8a14-ac60b6e9951e None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:49.736026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2306e8c5-cb64-4644-8a14-ac60b6e9951e None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:49.736026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2306e8c5-cb64-4644-8a14-ac60b6e9951e None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:49.737067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 498/1023] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:49.738493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e30c543-cf06-4694-8768-441e021bb645 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:49.738768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e30c543-cf06-4694-8768-441e021bb645 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:49.739435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e30c543-cf06-4694-8768-441e021bb645 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:49.739991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2306e8c5-cb64-4644-8a14-ac60b6e9951e None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:49.740794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e30c543-cf06-4694-8768-441e021bb645 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:49.831231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 526/1024] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:27:49 2022] POST /identity/v3/services => generated 242 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:27:49.832588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecc6389-4cc5-4ced-bd09-1cf12e803e28 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:49.832871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecc6389-4cc5-4ced-bd09-1cf12e803e28 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:49.833148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecc6389-4cc5-4ced-bd09-1cf12e803e28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:49.851358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:49.851574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2306e8c5-cb64-4644-8a14-ac60b6e9951e None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:49.859701 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5ecc6389-4cc5-4ced-bd09-1cf12e803e28 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:49.861004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5ecc6389-4cc5-4ced-bd09-1cf12e803e28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:49.894506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3bd158-59f9-452a-9536-66fa7626b9d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:49.895140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3bd158-59f9-452a-9536-66fa7626b9d3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:49.895426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3bd158-59f9-452a-9536-66fa7626b9d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:49.895683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3bd158-59f9-452a-9536-66fa7626b9d3 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:49.896464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3bd158-59f9-452a-9536-66fa7626b9d3 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:49.896734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3bd158-59f9-452a-9536-66fa7626b9d3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:49.898330 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3bd158-59f9-452a-9536-66fa7626b9d3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:49.925622 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 499/1025] 10.222.0.65 () {56 vars in 1133 bytes} [Wed Aug 24 03:27:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:49.926673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3bd158-59f9-452a-9536-66fa7626b9d3 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:49.962185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 527/1026] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:49 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:49.989236 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:50.044944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3bd158-59f9-452a-9536-66fa7626b9d3 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:50.080902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19081e9-eaf4-4013-8e49-59d8a8744495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:50.081849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19081e9-eaf4-4013-8e49-59d8a8744495 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:50.082339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19081e9-eaf4-4013-8e49-59d8a8744495 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:50.082778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19081e9-eaf4-4013-8e49-59d8a8744495 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:50.152835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19081e9-eaf4-4013-8e49-59d8a8744495 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:50.154591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19081e9-eaf4-4013-8e49-59d8a8744495 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:50.200041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:50.222613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecc6389-4cc5-4ced-bd09-1cf12e803e28 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:50.262999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 500/1027] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:49 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 24 03:27:50.266288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19081e9-eaf4-4013-8e49-59d8a8744495 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:50.340129 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecc6389-4cc5-4ced-bd09-1cf12e803e28 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:50.374691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19081e9-eaf4-4013-8e49-59d8a8744495 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:50.376244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db70ffb-166e-4e11-b953-dc68b33598e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:50.376879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db70ffb-166e-4e11-b953-dc68b33598e3 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:50.377233 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db70ffb-166e-4e11-b953-dc68b33598e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:50.377616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db70ffb-166e-4e11-b953-dc68b33598e3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:50.378757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db70ffb-166e-4e11-b953-dc68b33598e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:50.380185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db70ffb-166e-4e11-b953-dc68b33598e3 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:50.382231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 528/1028] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:50.388436 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db70ffb-166e-4e11-b953-dc68b33598e3 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:50.393504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:50.419147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7b8c91-a86a-4257-86f1-2c709b011d88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:50.419784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7b8c91-a86a-4257-86f1-2c709b011d88 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:50.420061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7b8c91-a86a-4257-86f1-2c709b011d88 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:50.420323 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7b8c91-a86a-4257-86f1-2c709b011d88 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:50.431441 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7b8c91-a86a-4257-86f1-2c709b011d88 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:50.432878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7b8c91-a86a-4257-86f1-2c709b011d88 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:50.435712 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-ed7b8c91-a86a-4257-86f1-2c709b011d88 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:50.448345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:50.449316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:50.450307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:50.450307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:50.450307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:50.450307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 501/1029] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:50.455044 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7b8c91-a86a-4257-86f1-2c709b011d88 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:50.502911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db70ffb-166e-4e11-b953-dc68b33598e3 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:50.539044 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be138028-bca6-4d2b-bbad-0834ece5f17f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:50.539651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be138028-bca6-4d2b-bbad-0834ece5f17f None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:50.539939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be138028-bca6-4d2b-bbad-0834ece5f17f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:50.540209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be138028-bca6-4d2b-bbad-0834ece5f17f None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:50.540863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be138028-bca6-4d2b-bbad-0834ece5f17f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:50.542323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be138028-bca6-4d2b-bbad-0834ece5f17f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:50.575195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7b8c91-a86a-4257-86f1-2c709b011d88 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:50.609681 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be34726-9d8b-473f-a457-30aa2596a681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:50.610345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be34726-9d8b-473f-a457-30aa2596a681 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:50.610604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be34726-9d8b-473f-a457-30aa2596a681 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:50.610869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be34726-9d8b-473f-a457-30aa2596a681 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:50.611528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be34726-9d8b-473f-a457-30aa2596a681 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:50.611825 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be34726-9d8b-473f-a457-30aa2596a681 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:50.613238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be34726-9d8b-473f-a457-30aa2596a681 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:50.614878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 529/1030] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:27:50 2022] POST /identity/v3/projects => generated 311 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:27:50.635903 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 502/1031] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:50.637273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be138028-bca6-4d2b-bbad-0834ece5f17f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:50.698957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:50.745981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be138028-bca6-4d2b-bbad-0834ece5f17f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:50.782857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b62abcc-1521-4078-8405-ea3aa544dc26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:50.784054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b62abcc-1521-4078-8405-ea3aa544dc26 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:50.784331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b62abcc-1521-4078-8405-ea3aa544dc26 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:50.784601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b62abcc-1521-4078-8405-ea3aa544dc26 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:50.791388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b62abcc-1521-4078-8405-ea3aa544dc26 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:50.792827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b62abcc-1521-4078-8405-ea3aa544dc26 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-7b62abcc-1521-4078-8405-ea3aa544dc26 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:50.804073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:50.804966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:50.805901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:50.805901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:50.805901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:50.805901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:50.805901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:50.805901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:50.805901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:50.805901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:50.805901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 530/1032] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:50.811156 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be34726-9d8b-473f-a457-30aa2596a681 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:50.877950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:50.923358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be34726-9d8b-473f-a457-30aa2596a681 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:50.959787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313c14e2-c8d7-40c0-96fc-068dc5440cbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:50.960455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313c14e2-c8d7-40c0-96fc-068dc5440cbf None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:50.960735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313c14e2-c8d7-40c0-96fc-068dc5440cbf None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:50.961007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313c14e2-c8d7-40c0-96fc-068dc5440cbf None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:50.961703 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313c14e2-c8d7-40c0-96fc-068dc5440cbf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:50.962005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313c14e2-c8d7-40c0-96fc-068dc5440cbf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:50.963521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313c14e2-c8d7-40c0-96fc-068dc5440cbf None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:50.981312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 503/1033] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:50.982511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b62abcc-1521-4078-8405-ea3aa544dc26 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:51.082872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b62abcc-1521-4078-8405-ea3aa544dc26 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:51.099281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:51.118676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d08539-bfb7-4056-96e3-d49a89887f57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:51.119293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d08539-bfb7-4056-96e3-d49a89887f57 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:51.119574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d08539-bfb7-4056-96e3-d49a89887f57 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:51.119844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d08539-bfb7-4056-96e3-d49a89887f57 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:51.120624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d08539-bfb7-4056-96e3-d49a89887f57 None admin] RBAC: Adding query filter params (role.id=1489a00dd22146ecb6161ff7d549f95c, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=e5896018712d4b82a781365e478ee855) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:51.120929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d08539-bfb7-4056-96e3-d49a89887f57 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:51.122509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d08539-bfb7-4056-96e3-d49a89887f57 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:51.134187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 531/1034] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:50 2022] GET /identity/v3/role_assignments?user.id=e5896018712d4b82a781365e478ee855&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=1489a00dd22146ecb6161ff7d549f95c => generated 259 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:51.798061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d044df-68dd-43f2-bba6-8cdd5246573a None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:51.798593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d044df-68dd-43f2-bba6-8cdd5246573a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:51.798593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d044df-68dd-43f2-bba6-8cdd5246573a None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:51.800573 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 504/1035] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:51.805572 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d8f954-0450-4430-94a7-cf246ff6bc35 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:51.805969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d8f954-0450-4430-94a7-cf246ff6bc35 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:51.806206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d8f954-0450-4430-94a7-cf246ff6bc35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:51.825415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-01d8f954-0450-4430-94a7-cf246ff6bc35 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:51.826634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-01d8f954-0450-4430-94a7-cf246ff6bc35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:51.891046 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 532/1036] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:51 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:52.163433 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d044df-68dd-43f2-bba6-8cdd5246573a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:52.269233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d044df-68dd-43f2-bba6-8cdd5246573a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:52.286889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:52.304692 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df615b2b-1dfa-4a47-b2d2-90ec1e4dd152 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:52.305321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df615b2b-1dfa-4a47-b2d2-90ec1e4dd152 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:52.305598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df615b2b-1dfa-4a47-b2d2-90ec1e4dd152 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:52.305865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df615b2b-1dfa-4a47-b2d2-90ec1e4dd152 None admin] PATH_INFO: `/v3/services/image` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:52.306474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df615b2b-1dfa-4a47-b2d2-90ec1e4dd152 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:52.307826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df615b2b-1dfa-4a47-b2d2-90ec1e4dd152 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-df615b2b-1dfa-4a47-b2d2-90ec1e4dd152 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:52.317615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:52.318684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:52.319557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 505/1037] 10.222.0.65 () {56 vars in 1105 bytes} [Wed Aug 24 03:27:52 2022] GET /identity/v3/services/image => generated 86 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:52.326553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d8f954-0450-4430-94a7-cf246ff6bc35 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:52.432777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d8f954-0450-4430-94a7-cf246ff6bc35 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:52.444721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:52.468446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7094e76-d438-4011-8fc4-b4218ed45af7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:52.469055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7094e76-d438-4011-8fc4-b4218ed45af7 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:52.469329 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7094e76-d438-4011-8fc4-b4218ed45af7 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:52.469600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7094e76-d438-4011-8fc4-b4218ed45af7 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:52.470659 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7094e76-d438-4011-8fc4-b4218ed45af7 None admin] RBAC: Adding query filter params (name=image) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:52.471637 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7094e76-d438-4011-8fc4-b4218ed45af7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:52.473608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7094e76-d438-4011-8fc4-b4218ed45af7 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:52.478725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 533/1038] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:52 2022] GET /identity/v3/services?name=image => generated 122 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:52.484490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df615b2b-1dfa-4a47-b2d2-90ec1e4dd152 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:52.587152 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df615b2b-1dfa-4a47-b2d2-90ec1e4dd152 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:52.594648 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e9ae88-006d-42af-b700-25c447e117fd None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:52.594934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e9ae88-006d-42af-b700-25c447e117fd None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:52.595205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e9ae88-006d-42af-b700-25c447e117fd None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:52.596812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 534/1039] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:52.601828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845b86b2-b677-41a9-acce-1cf291b4e90c None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:52.602114 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845b86b2-b677-41a9-acce-1cf291b4e90c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:52.602416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845b86b2-b677-41a9-acce-1cf291b4e90c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:52.621068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-845b86b2-b677-41a9-acce-1cf291b4e90c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:52.622207 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96a7808-a80e-4b2f-a084-04838282e4bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:52.622358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-845b86b2-b677-41a9-acce-1cf291b4e90c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:52.622814 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a7808-a80e-4b2f-a084-04838282e4bb None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:52.623084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a7808-a80e-4b2f-a084-04838282e4bb None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:52.623348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a7808-a80e-4b2f-a084-04838282e4bb None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:52.623963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96a7808-a80e-4b2f-a084-04838282e4bb None admin] RBAC: Adding query filter params (type=image) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:52.624260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96a7808-a80e-4b2f-a084-04838282e4bb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:52.625594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96a7808-a80e-4b2f-a084-04838282e4bb None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:52.631822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 506/1040] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:52 2022] GET /identity/v3/services?type=image => generated 350 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:52.637630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96a7808-a80e-4b2f-a084-04838282e4bb None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:52.683862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:52.698605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 535/1041] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:52 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:52.742215 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96a7808-a80e-4b2f-a084-04838282e4bb None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:52.779123 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ca4cb0-a8dc-4712-8bf0-2b8861c1306d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:52.779815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ca4cb0-a8dc-4712-8bf0-2b8861c1306d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:52.780158 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ca4cb0-a8dc-4712-8bf0-2b8861c1306d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:52.780492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ca4cb0-a8dc-4712-8bf0-2b8861c1306d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:52.781282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ca4cb0-a8dc-4712-8bf0-2b8861c1306d None admin] RBAC: Adding query filter params (interface=public, service_id=af9de7246b6242fead8d79bad51ce533, region_id=RegionOne) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:52.781679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ca4cb0-a8dc-4712-8bf0-2b8861c1306d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:52.783121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ca4cb0-a8dc-4712-8bf0-2b8861c1306d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:52.793162 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 507/1042] 10.222.0.65 () {56 vars in 1251 bytes} [Wed Aug 24 03:27:52 2022] GET /identity/v3/endpoints?service_id=af9de7246b6242fead8d79bad51ce533&interface=public®ion_id=RegionOne => generated 194 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:52.794306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845b86b2-b677-41a9-acce-1cf291b4e90c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:52.894397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845b86b2-b677-41a9-acce-1cf291b4e90c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:52.896718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:52.928765 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6575bda6-a58e-4098-bd88-c7731afdd390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:52.929386 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6575bda6-a58e-4098-bd88-c7731afdd390 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:52.929661 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6575bda6-a58e-4098-bd88-c7731afdd390 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:52.929930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6575bda6-a58e-4098-bd88-c7731afdd390 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:52.930552 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6575bda6-a58e-4098-bd88-c7731afdd390 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:52.931899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6575bda6-a58e-4098-bd88-c7731afdd390 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:52.942491 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 536/1043] 10.222.0.65 () {56 vars in 1087 bytes} [Wed Aug 24 03:27:52 2022] GET /identity/v3/services => generated 1957 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:27:52.973967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ca4cb0-a8dc-4712-8bf0-2b8861c1306d None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:53.026308 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd458b6-f4aa-4b1e-ae3d-fd7a725142b2 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:53.026647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd458b6-f4aa-4b1e-ae3d-fd7a725142b2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:53.026861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd458b6-f4aa-4b1e-ae3d-fd7a725142b2 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:53.028204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 537/1044] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:53.033686 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a91007-4e1b-410d-9498-1e11f8aa82c6 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:53.033987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a91007-4e1b-410d-9498-1e11f8aa82c6 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:53.034262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a91007-4e1b-410d-9498-1e11f8aa82c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:53.053212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-90a91007-4e1b-410d-9498-1e11f8aa82c6 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:53.054472 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-90a91007-4e1b-410d-9498-1e11f8aa82c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:53.087135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ca4cb0-a8dc-4712-8bf0-2b8861c1306d None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:53.097707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:53.129411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b63db7c-f8f2-40f1-b064-b78d20892d09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:53.130037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b63db7c-f8f2-40f1-b064-b78d20892d09 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:53.130341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b63db7c-f8f2-40f1-b064-b78d20892d09 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:53.130602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b63db7c-f8f2-40f1-b064-b78d20892d09 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:53.131988 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b63db7c-f8f2-40f1-b064-b78d20892d09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:53.133720 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b63db7c-f8f2-40f1-b064-b78d20892d09 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:53.135505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 508/1045] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:53.141262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 538/1046] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:53 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:53.141467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b63db7c-f8f2-40f1-b064-b78d20892d09 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:53.241508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b63db7c-f8f2-40f1-b064-b78d20892d09 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:53.276554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f28aa82-4e77-428e-b1d0-73cc1120cc54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:53.277161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f28aa82-4e77-428e-b1d0-73cc1120cc54 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:53.277438 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f28aa82-4e77-428e-b1d0-73cc1120cc54 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:53.277706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f28aa82-4e77-428e-b1d0-73cc1120cc54 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:53.278395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f28aa82-4e77-428e-b1d0-73cc1120cc54 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:53.279823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f28aa82-4e77-428e-b1d0-73cc1120cc54 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:53.379714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 509/1047] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:27:53 2022] POST /identity/v3/users => generated 285 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:27:53.414301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a91007-4e1b-410d-9498-1e11f8aa82c6 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:53.520732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:53.525025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a91007-4e1b-410d-9498-1e11f8aa82c6 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:53.561239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8049d12-4f71-401c-8b8c-984ccf4d77e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:53.561873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8049d12-4f71-401c-8b8c-984ccf4d77e7 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:53.562188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8049d12-4f71-401c-8b8c-984ccf4d77e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:53.562456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8049d12-4f71-401c-8b8c-984ccf4d77e7 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:53.573606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8049d12-4f71-401c-8b8c-984ccf4d77e7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-b8049d12-4f71-401c-8b8c-984ccf4d77e7 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:53.578689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:53.579613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:53.580524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:53.580524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:53.580524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:53.580524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:53.580524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:53.580524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:53.580524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:53.580524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:53.580524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:53.580524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:53.580524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 03:27:53.580524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:53.582551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 539/1048] 10.222.0.65 () {56 vars in 1120 bytes} [Wed Aug 24 03:27:53 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:53.588785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f28aa82-4e77-428e-b1d0-73cc1120cc54 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:53.622434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:53.694263 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f28aa82-4e77-428e-b1d0-73cc1120cc54 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:53.729523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70f303b-0afa-4b7f-977c-75748f639669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:53.730125 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70f303b-0afa-4b7f-977c-75748f639669 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:53.730441 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70f303b-0afa-4b7f-977c-75748f639669 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:53.730709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70f303b-0afa-4b7f-977c-75748f639669 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:53.731396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70f303b-0afa-4b7f-977c-75748f639669 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:53.731695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70f303b-0afa-4b7f-977c-75748f639669 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:53.733140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70f303b-0afa-4b7f-977c-75748f639669 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:53.746117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 510/1049] 10.222.0.65 () {56 vars in 1115 bytes} [Wed Aug 24 03:27:53 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:53.747739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8049d12-4f71-401c-8b8c-984ccf4d77e7 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:53.850477 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8049d12-4f71-401c-8b8c-984ccf4d77e7 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:53.884212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55c09ec-ddd5-40da-8954-422008ded491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:53.884838 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55c09ec-ddd5-40da-8954-422008ded491 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:53.885117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55c09ec-ddd5-40da-8954-422008ded491 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:53.885389 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55c09ec-ddd5-40da-8954-422008ded491 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:53.896787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55c09ec-ddd5-40da-8954-422008ded491 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:53.898267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55c09ec-ddd5-40da-8954-422008ded491 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-a55c09ec-ddd5-40da-8954-422008ded491 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:53.912498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:53.913481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:53.919055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:53.919055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:53.919055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:53.919055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 540/1050] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:53.920965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70f303b-0afa-4b7f-977c-75748f639669 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:54.015897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:54.023179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70f303b-0afa-4b7f-977c-75748f639669 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:54.059598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0996be94-7e0c-46a5-b707-bd90c05333c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:54.060206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0996be94-7e0c-46a5-b707-bd90c05333c2 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:54.060482 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0996be94-7e0c-46a5-b707-bd90c05333c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:54.060752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0996be94-7e0c-46a5-b707-bd90c05333c2 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:54.061386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0996be94-7e0c-46a5-b707-bd90c05333c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:54.061681 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0996be94-7e0c-46a5-b707-bd90c05333c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:54.063057 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0996be94-7e0c-46a5-b707-bd90c05333c2 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:54.079112 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 511/1051] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:54.081211 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55c09ec-ddd5-40da-8954-422008ded491 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:54.094071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:54.181271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55c09ec-ddd5-40da-8954-422008ded491 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:54.213927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468b94fb-b827-4a3e-8985-78f86bbe3049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:54.214574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468b94fb-b827-4a3e-8985-78f86bbe3049 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:54.214850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468b94fb-b827-4a3e-8985-78f86bbe3049 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:54.215117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468b94fb-b827-4a3e-8985-78f86bbe3049 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:54.223066 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468b94fb-b827-4a3e-8985-78f86bbe3049 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:54.224523 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468b94fb-b827-4a3e-8985-78f86bbe3049 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-468b94fb-b827-4a3e-8985-78f86bbe3049 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:54.235210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:54.236137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:54.241948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:54.241948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:54.241948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:54.241948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:54.241948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:54.241948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:54.241948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:54.241948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:54.241948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:54.241948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:54.241948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 541/1052] 10.222.0.65 () {56 vars in 1093 bytes} [Wed Aug 24 03:27:54 2022] GET /identity/v3/users/nova => generated 82 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:54.244122 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0996be94-7e0c-46a5-b707-bd90c05333c2 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:54.348291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0996be94-7e0c-46a5-b707-bd90c05333c2 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:54.381679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acf43bc-01a4-49cc-bde3-4d62e5deb7b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:54.382311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acf43bc-01a4-49cc-bde3-4d62e5deb7b8 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:54.382589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acf43bc-01a4-49cc-bde3-4d62e5deb7b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:54.382858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acf43bc-01a4-49cc-bde3-4d62e5deb7b8 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:54.383579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acf43bc-01a4-49cc-bde3-4d62e5deb7b8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:54.383878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acf43bc-01a4-49cc-bde3-4d62e5deb7b8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:54.385297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acf43bc-01a4-49cc-bde3-4d62e5deb7b8 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:54.406360 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 512/1053] 10.222.0.65 () {56 vars in 1133 bytes} [Wed Aug 24 03:27:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:54.414234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468b94fb-b827-4a3e-8985-78f86bbe3049 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:54.487905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:54.519618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468b94fb-b827-4a3e-8985-78f86bbe3049 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:54.557161 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ba1ae-2173-450b-a55c-2d42460c9da9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:54.557800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ba1ae-2173-450b-a55c-2d42460c9da9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:54.558081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ba1ae-2173-450b-a55c-2d42460c9da9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:54.558383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ba1ae-2173-450b-a55c-2d42460c9da9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:54.569377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096ba1ae-2173-450b-a55c-2d42460c9da9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:54.570914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096ba1ae-2173-450b-a55c-2d42460c9da9 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-096ba1ae-2173-450b-a55c-2d42460c9da9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:54.584912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:54.585909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:54.591610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:54.591610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:54.591610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:54.591610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 542/1054] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:54.593425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acf43bc-01a4-49cc-bde3-4d62e5deb7b8 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:54.608777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:54.695083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acf43bc-01a4-49cc-bde3-4d62e5deb7b8 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:54.729413 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0241d9-a6f1-4078-abd0-fcb93aee7e75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:54.730020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0241d9-a6f1-4078-abd0-fcb93aee7e75 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:54.730329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0241d9-a6f1-4078-abd0-fcb93aee7e75 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:54.730597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0241d9-a6f1-4078-abd0-fcb93aee7e75 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:54.731371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0241d9-a6f1-4078-abd0-fcb93aee7e75 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:54.731714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0241d9-a6f1-4078-abd0-fcb93aee7e75 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:54.733073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0241d9-a6f1-4078-abd0-fcb93aee7e75 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:54.749533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 513/1055] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:54.750980 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ba1ae-2173-450b-a55c-2d42460c9da9 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:54.858214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ba1ae-2173-450b-a55c-2d42460c9da9 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:54.882490 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:54.889616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1723b907-d070-43d9-98c0-3bcfc3841025 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:54.889905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1723b907-d070-43d9-98c0-3bcfc3841025 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:54.890202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1723b907-d070-43d9-98c0-3bcfc3841025 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:54.894191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 514/1056] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:54.897383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e0ee18-bff7-44e9-8355-ffbb04f749be None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:54.897664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9558c8a5-1f31-4517-ab0a-ec01ad7e751c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:54.898351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9558c8a5-1f31-4517-ab0a-ec01ad7e751c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:54.898613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e0ee18-bff7-44e9-8355-ffbb04f749be None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:54.898714 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9558c8a5-1f31-4517-ab0a-ec01ad7e751c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:54.898895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9558c8a5-1f31-4517-ab0a-ec01ad7e751c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:54.899338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e0ee18-bff7-44e9-8355-ffbb04f749be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:54.905851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9558c8a5-1f31-4517-ab0a-ec01ad7e751c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:54.907339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9558c8a5-1f31-4517-ab0a-ec01ad7e751c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:54.918649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-53e0ee18-bff7-44e9-8355-ffbb04f749be None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-9558c8a5-1f31-4517-ab0a-ec01ad7e751c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:54.919150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:54.920056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:54.921297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:54.921297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:54.921297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:54.921297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:54.921297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:54.921297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:54.921297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:54.921297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:54.921297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 543/1057] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:54.923022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-53e0ee18-bff7-44e9-8355-ffbb04f749be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:54.926023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9558c8a5-1f31-4517-ab0a-ec01ad7e751c None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:55.007451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 515/1058] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:54 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:55.029619 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9558c8a5-1f31-4517-ab0a-ec01ad7e751c None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:55.063866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1399c834-4651-4b53-bbe4-1cfd4a483980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:55.064999 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1399c834-4651-4b53-bbe4-1cfd4a483980 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:55.065278 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1399c834-4651-4b53-bbe4-1cfd4a483980 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:55.065550 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1399c834-4651-4b53-bbe4-1cfd4a483980 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:55.066260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1399c834-4651-4b53-bbe4-1cfd4a483980 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:55.066562 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1399c834-4651-4b53-bbe4-1cfd4a483980 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:55.067982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1399c834-4651-4b53-bbe4-1cfd4a483980 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:55.079924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 544/1059] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:55.085956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e0ee18-bff7-44e9-8355-ffbb04f749be None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:55.088815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:55.185883 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e0ee18-bff7-44e9-8355-ffbb04f749be None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:55.219626 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f048606-4d9a-4ecf-8353-1f32fde60db1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:55.220271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f048606-4d9a-4ecf-8353-1f32fde60db1 None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:55.220542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f048606-4d9a-4ecf-8353-1f32fde60db1 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:55.221549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f048606-4d9a-4ecf-8353-1f32fde60db1 None admin] PATH_INFO: `/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/e5896018712d4b82a781365e478ee855/roles/1489a00dd22146ecb6161ff7d549f95c` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:55.223497 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f048606-4d9a-4ecf-8353-1f32fde60db1 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c653d5fa4304f298a295ce635efd5e6, user_id=e5896018712d4b82a781365e478ee855, role_id=1489a00dd22146ecb6161ff7d549f95c)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:55.224922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f048606-4d9a-4ecf-8353-1f32fde60db1 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:55.279753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1399c834-4651-4b53-bbe4-1cfd4a483980 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:55.300472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 516/1060] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:27:55 2022] PUT /identity/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/e5896018712d4b82a781365e478ee855/roles/1489a00dd22146ecb6161ff7d549f95c => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:27:55.333619 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:55.384063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e7699d-29bb-45dc-88b9-357ab8828f7c None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:55.384401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e7699d-29bb-45dc-88b9-357ab8828f7c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:55.384597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e7699d-29bb-45dc-88b9-357ab8828f7c None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:55.386003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 517/1061] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:55.391546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7667d6e1-ec2c-4257-8b61-75acbb69ce4f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:55.391823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7667d6e1-ec2c-4257-8b61-75acbb69ce4f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:55.392090 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7667d6e1-ec2c-4257-8b61-75acbb69ce4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:55.399482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1399c834-4651-4b53-bbe4-1cfd4a483980 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:55.412789 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-7667d6e1-ec2c-4257-8b61-75acbb69ce4f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:55.413865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7667d6e1-ec2c-4257-8b61-75acbb69ce4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:55.454679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feebc619-e168-49e2-8b6d-b23b5acda13b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:55.455850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feebc619-e168-49e2-8b6d-b23b5acda13b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:55.456390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feebc619-e168-49e2-8b6d-b23b5acda13b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:55.456518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feebc619-e168-49e2-8b6d-b23b5acda13b None admin] PATH_INFO: `/v3/services/image` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:55.457075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feebc619-e168-49e2-8b6d-b23b5acda13b None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:55.458457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feebc619-e168-49e2-8b6d-b23b5acda13b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-feebc619-e168-49e2-8b6d-b23b5acda13b None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:55.467575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:55.468631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:55.469739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 545/1062] 10.222.0.65 () {56 vars in 1105 bytes} [Wed Aug 24 03:27:55 2022] GET /identity/v3/services/image => generated 86 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:55.476111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feebc619-e168-49e2-8b6d-b23b5acda13b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:55.483524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:55.496117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 518/1063] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:55 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:27:55.581557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feebc619-e168-49e2-8b6d-b23b5acda13b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:55.589335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:55.617768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1da1a53-e770-4331-8ca4-6c02d17b0233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:55.618548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1da1a53-e770-4331-8ca4-6c02d17b0233 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:55.618927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1da1a53-e770-4331-8ca4-6c02d17b0233 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:55.619299 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1da1a53-e770-4331-8ca4-6c02d17b0233 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:55.620029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1da1a53-e770-4331-8ca4-6c02d17b0233 None admin] RBAC: Adding query filter params (name=image) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:55.620430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1da1a53-e770-4331-8ca4-6c02d17b0233 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:55.621882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1da1a53-e770-4331-8ca4-6c02d17b0233 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:55.631745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 546/1064] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:55 2022] GET /identity/v3/services?name=image => generated 122 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:55.633086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7667d6e1-ec2c-4257-8b61-75acbb69ce4f None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:55.735911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7667d6e1-ec2c-4257-8b61-75acbb69ce4f None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:55.767209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1da1a53-e770-4331-8ca4-6c02d17b0233 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:55.772290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0414f2a-70ad-45e3-aab7-e7d7b001ed7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:55.773031 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0414f2a-70ad-45e3-aab7-e7d7b001ed7c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:55.773402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0414f2a-70ad-45e3-aab7-e7d7b001ed7c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:55.773769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0414f2a-70ad-45e3-aab7-e7d7b001ed7c None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:55.774537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0414f2a-70ad-45e3-aab7-e7d7b001ed7c None admin] RBAC: Adding query filter params (type=image) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:55.774952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0414f2a-70ad-45e3-aab7-e7d7b001ed7c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:55.776401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0414f2a-70ad-45e3-aab7-e7d7b001ed7c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:55.782799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 519/1065] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:55 2022] GET /identity/v3/services?type=image => generated 350 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:55.789174 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0414f2a-70ad-45e3-aab7-e7d7b001ed7c None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:55.848843 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:55.874707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1da1a53-e770-4331-8ca4-6c02d17b0233 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:55.894181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0414f2a-70ad-45e3-aab7-e7d7b001ed7c None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:55.912802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36edb2a-1378-407d-bf8e-36543c4429b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:55.913426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36edb2a-1378-407d-bf8e-36543c4429b6 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:55.913696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36edb2a-1378-407d-bf8e-36543c4429b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:55.913959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36edb2a-1378-407d-bf8e-36543c4429b6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:55.924765 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36edb2a-1378-407d-bf8e-36543c4429b6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:55.928301 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8004635-98af-4859-8957-d9ec9e25bbaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:55.928906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8004635-98af-4859-8957-d9ec9e25bbaa None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:55.929181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8004635-98af-4859-8957-d9ec9e25bbaa None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-b36edb2a-1378-407d-bf8e-36543c4429b6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:55.929391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:55.930327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:55.931274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:55.931274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:55.931274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:55.931274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:55.931274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:55.931274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:55.931274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:55.931274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:55.931274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:55.931274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:55.931274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:55.931274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:55.932398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8004635-98af-4859-8957-d9ec9e25bbaa None admin] PATH_INFO: `/v3/endpoints` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:55.932398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8004635-98af-4859-8957-d9ec9e25bbaa None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:55.932398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8004635-98af-4859-8957-d9ec9e25bbaa None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:55.935560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 547/1066] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:55.936295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36edb2a-1378-407d-bf8e-36543c4429b6 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:55.967773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:55.996780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 520/1067] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:27:55 2022] POST /identity/v3/endpoints => generated 327 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:27:56.043600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36edb2a-1378-407d-bf8e-36543c4429b6 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:56.089098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef01c6b-4587-42c1-8110-84a78da7ae13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:56.089709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef01c6b-4587-42c1-8110-84a78da7ae13 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:56.089987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef01c6b-4587-42c1-8110-84a78da7ae13 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:56.090288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef01c6b-4587-42c1-8110-84a78da7ae13 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:56.090958 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef01c6b-4587-42c1-8110-84a78da7ae13 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:56.091257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef01c6b-4587-42c1-8110-84a78da7ae13 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:56.092636 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef01c6b-4587-42c1-8110-84a78da7ae13 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:56.106544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 548/1068] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:27:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:56.107679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8004635-98af-4859-8957-d9ec9e25bbaa None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:56.219305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8004635-98af-4859-8957-d9ec9e25bbaa None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:56.232564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:56.255218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0d5df6-0a04-47c0-851a-d2c1d6abbbb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:56.256417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d5df6-0a04-47c0-851a-d2c1d6abbbb0 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:56.256869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d5df6-0a04-47c0-851a-d2c1d6abbbb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:56.257314 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d5df6-0a04-47c0-851a-d2c1d6abbbb0 None admin] PATH_INFO: `/v3/users/2cecc223594344cc9c0c1c2230d125e5` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:56.272173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0d5df6-0a04-47c0-851a-d2c1d6abbbb0 None admin] RBAC: Authorizing `identity:get_user(user_id=2cecc223594344cc9c0c1c2230d125e5)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:56.273890 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0d5df6-0a04-47c0-851a-d2c1d6abbbb0 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:56.275777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 521/1069] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:27:56 2022] GET /identity/v3/users/2cecc223594344cc9c0c1c2230d125e5 => generated 285 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:56.282029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef01c6b-4587-42c1-8110-84a78da7ae13 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:56.376036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:56.389563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef01c6b-4587-42c1-8110-84a78da7ae13 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:56.432270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29733030-48d6-4e0a-8ec9-fb19068e9894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:56.432880 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29733030-48d6-4e0a-8ec9-fb19068e9894 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:56.433160 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29733030-48d6-4e0a-8ec9-fb19068e9894 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:56.433430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29733030-48d6-4e0a-8ec9-fb19068e9894 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:56.504390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29733030-48d6-4e0a-8ec9-fb19068e9894 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:56.506091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29733030-48d6-4e0a-8ec9-fb19068e9894 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:56.590704 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:56.608872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 549/1070] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:56 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:27:56.610446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0d5df6-0a04-47c0-851a-d2c1d6abbbb0 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:56.712079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0d5df6-0a04-47c0-851a-d2c1d6abbbb0 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:56.746455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f38b037-f8a0-4204-b10a-c8b9b419e0f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:56.747084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f38b037-f8a0-4204-b10a-c8b9b419e0f3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:56.747354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f38b037-f8a0-4204-b10a-c8b9b419e0f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:56.747628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f38b037-f8a0-4204-b10a-c8b9b419e0f3 None admin] PATH_INFO: `/v3/projects/288ff8851fff49c1b5c75e576bee9d4b` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:56.748759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f38b037-f8a0-4204-b10a-c8b9b419e0f3 None admin] RBAC: Authorizing `identity:get_project(project_id=288ff8851fff49c1b5c75e576bee9d4b)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:56.750199 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f38b037-f8a0-4204-b10a-c8b9b419e0f3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:56.757106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 522/1071] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:27:56 2022] GET /identity/v3/projects/288ff8851fff49c1b5c75e576bee9d4b => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:56.758662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29733030-48d6-4e0a-8ec9-fb19068e9894 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:56.857434 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29733030-48d6-4e0a-8ec9-fb19068e9894 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:56.892259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf41f41-b34a-433f-98e2-38a82588bc90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:56.893614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf41f41-b34a-433f-98e2-38a82588bc90 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:56.893888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf41f41-b34a-433f-98e2-38a82588bc90 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:56.894183 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf41f41-b34a-433f-98e2-38a82588bc90 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:56.894911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf41f41-b34a-433f-98e2-38a82588bc90 None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=288ff8851fff49c1b5c75e576bee9d4b, user.id=2cecc223594344cc9c0c1c2230d125e5) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:56.895217 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf41f41-b34a-433f-98e2-38a82588bc90 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:56.896651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf41f41-b34a-433f-98e2-38a82588bc90 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:56.910248 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 550/1072] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:56 2022] GET /identity/v3/role_assignments?user.id=2cecc223594344cc9c0c1c2230d125e5&scope.project.id=288ff8851fff49c1b5c75e576bee9d4b&role.id=43ca25467bc6484da3a50508efb79230 => generated 259 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:57.243057 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecf6cff-93c1-46c2-9526-60d44b7e46d0 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:57.243574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecf6cff-93c1-46c2-9526-60d44b7e46d0 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:57.243574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecf6cff-93c1-46c2-9526-60d44b7e46d0 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:57.244943 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 523/1073] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:57.250275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5c9da2-51ed-410a-a849-801917a2bb4f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:57.250712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5c9da2-51ed-410a-a849-801917a2bb4f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:57.250883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5c9da2-51ed-410a-a849-801917a2bb4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:57.269901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-9c5c9da2-51ed-410a-a849-801917a2bb4f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:57.271146 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9c5c9da2-51ed-410a-a849-801917a2bb4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:57.322293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:57.342128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 551/1074] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:57.608219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecf6cff-93c1-46c2-9526-60d44b7e46d0 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:57.652277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:57.713063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecf6cff-93c1-46c2-9526-60d44b7e46d0 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:57.747233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16478cfe-48eb-4f31-bd6a-bb2feb730e27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:57.747859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16478cfe-48eb-4f31-bd6a-bb2feb730e27 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:57.748136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16478cfe-48eb-4f31-bd6a-bb2feb730e27 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:57.748406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16478cfe-48eb-4f31-bd6a-bb2feb730e27 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:57.759378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16478cfe-48eb-4f31-bd6a-bb2feb730e27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-16478cfe-48eb-4f31-bd6a-bb2feb730e27 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:57.764047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:57.764959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:57.765818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:57.765818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:57.765818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:57.765818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:57.765818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:57.765818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:57.765818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:57.765818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:57.765818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:57.765818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:57.765818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 24 03:27:57.765818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:57.767182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 524/1075] 10.222.0.65 () {56 vars in 1120 bytes} [Wed Aug 24 03:27:57 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:57.771157 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5c9da2-51ed-410a-a849-801917a2bb4f None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:57.871095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5c9da2-51ed-410a-a849-801917a2bb4f None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:57.905830 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82c8e1e-d95c-4b2b-9801-bf3f2070bdc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:57.906469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82c8e1e-d95c-4b2b-9801-bf3f2070bdc9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:57.906753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82c8e1e-d95c-4b2b-9801-bf3f2070bdc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:57.907022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82c8e1e-d95c-4b2b-9801-bf3f2070bdc9 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:57.907686 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82c8e1e-d95c-4b2b-9801-bf3f2070bdc9 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:57.907988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82c8e1e-d95c-4b2b-9801-bf3f2070bdc9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:57.909389 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82c8e1e-d95c-4b2b-9801-bf3f2070bdc9 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:57.917991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 552/1076] 10.222.0.65 () {56 vars in 1115 bytes} [Wed Aug 24 03:27:57 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:57.923667 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16478cfe-48eb-4f31-bd6a-bb2feb730e27 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:57.952955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0d4c30-4c26-47b9-8701-b4bbbda5e7ef None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:57.953240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0d4c30-4c26-47b9-8701-b4bbbda5e7ef None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:57.953515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0d4c30-4c26-47b9-8701-b4bbbda5e7ef None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:57.955181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 553/1077] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:27:57.960715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6383ce66-1288-4503-aa60-8cbf9fa9a752 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:57.961022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6383ce66-1288-4503-aa60-8cbf9fa9a752 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:57.961301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6383ce66-1288-4503-aa60-8cbf9fa9a752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:57.979860 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-6383ce66-1288-4503-aa60-8cbf9fa9a752 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:57.981069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6383ce66-1288-4503-aa60-8cbf9fa9a752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:58.022885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:58.033043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:58.037982 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16478cfe-48eb-4f31-bd6a-bb2feb730e27 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:58.051329 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 554/1078] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:58.072142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4d0647-63c7-4372-8402-4c9daf016c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:58.072769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4d0647-63c7-4372-8402-4c9daf016c97 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:58.073048 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4d0647-63c7-4372-8402-4c9daf016c97 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:58.073320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4d0647-63c7-4372-8402-4c9daf016c97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:58.083249 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4d0647-63c7-4372-8402-4c9daf016c97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:58.084720 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4d0647-63c7-4372-8402-4c9daf016c97 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-3b4d0647-63c7-4372-8402-4c9daf016c97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:58.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:58.100999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:58.101932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:58.101932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:58.101932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:58.101932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 525/1079] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:58.106977 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6383ce66-1288-4503-aa60-8cbf9fa9a752 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:58.205652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6383ce66-1288-4503-aa60-8cbf9fa9a752 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:58.240955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fd73d0-6b5e-4b98-bf95-c5496595c306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:58.241567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fd73d0-6b5e-4b98-bf95-c5496595c306 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:58.241989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fd73d0-6b5e-4b98-bf95-c5496595c306 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:58.242207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fd73d0-6b5e-4b98-bf95-c5496595c306 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:58.242879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fd73d0-6b5e-4b98-bf95-c5496595c306 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:58.243181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fd73d0-6b5e-4b98-bf95-c5496595c306 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:58.244574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fd73d0-6b5e-4b98-bf95-c5496595c306 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:58.261352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 555/1080] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:58.262553 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4d0647-63c7-4372-8402-4c9daf016c97 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:58.321031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fd73d0-6b5e-4b98-bf95-c5496595c306 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:58.357255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:58.366925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4d0647-63c7-4372-8402-4c9daf016c97 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:58.402992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4558b53-95a6-49bc-990b-78ebc1944485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:58.403626 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4558b53-95a6-49bc-990b-78ebc1944485 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:58.403904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4558b53-95a6-49bc-990b-78ebc1944485 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:58.404176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4558b53-95a6-49bc-990b-78ebc1944485 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:58.412671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4558b53-95a6-49bc-990b-78ebc1944485 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:58.414114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4558b53-95a6-49bc-990b-78ebc1944485 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-b4558b53-95a6-49bc-990b-78ebc1944485 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:27:58.426742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:58.427726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:58.428688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:58.428688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:58.428688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:27:58.428688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:27:58.428688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:27:58.428688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:27:58.428688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:27:58.428688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:27:58.428688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 24 03:27:58.428688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:58.428688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 526/1081] 10.222.0.65 () {56 vars in 1093 bytes} [Wed Aug 24 03:27:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:58.433733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4558b53-95a6-49bc-990b-78ebc1944485 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:58.443216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fd73d0-6b5e-4b98-bf95-c5496595c306 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:58.447037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:58.482279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fea49a3-1b05-44cb-ac85-cdbd7adb2c50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:58.482748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fea49a3-1b05-44cb-ac85-cdbd7adb2c50 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:58.483027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fea49a3-1b05-44cb-ac85-cdbd7adb2c50 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:58.483461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fea49a3-1b05-44cb-ac85-cdbd7adb2c50 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:58.545068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4558b53-95a6-49bc-990b-78ebc1944485 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:58.557467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fea49a3-1b05-44cb-ac85-cdbd7adb2c50 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:58.558952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fea49a3-1b05-44cb-ac85-cdbd7adb2c50 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:58.580180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285d07e5-4767-4be8-9d62-07521f51a779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:58.580779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285d07e5-4767-4be8-9d62-07521f51a779 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:58.581052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285d07e5-4767-4be8-9d62-07521f51a779 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:58.581321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285d07e5-4767-4be8-9d62-07521f51a779 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:58.582043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285d07e5-4767-4be8-9d62-07521f51a779 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:58.582384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285d07e5-4767-4be8-9d62-07521f51a779 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:58.583783 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285d07e5-4767-4be8-9d62-07521f51a779 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:58.595475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:58.601187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 527/1082] 10.222.0.65 () {56 vars in 1133 bytes} [Wed Aug 24 03:27:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:58.606823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285d07e5-4767-4be8-9d62-07521f51a779 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:58.668075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 556/1083] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 24 03:27:58.673395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fea49a3-1b05-44cb-ac85-cdbd7adb2c50 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:58.678891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:58.713641 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285d07e5-4767-4be8-9d62-07521f51a779 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:58.753894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a92fa0-1c8f-4002-a2e3-64d747528ce8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:58.754737 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a92fa0-1c8f-4002-a2e3-64d747528ce8 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:58.755039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a92fa0-1c8f-4002-a2e3-64d747528ce8 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:58.755406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a92fa0-1c8f-4002-a2e3-64d747528ce8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:58.777300 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fea49a3-1b05-44cb-ac85-cdbd7adb2c50 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:58.811294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283312ff-7121-4662-bf62-3f184cbc567c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:58.812332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283312ff-7121-4662-bf62-3f184cbc567c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:58.812617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283312ff-7121-4662-bf62-3f184cbc567c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:58.812890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283312ff-7121-4662-bf62-3f184cbc567c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:58.815458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:58.823544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283312ff-7121-4662-bf62-3f184cbc567c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:58.824982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283312ff-7121-4662-bf62-3f184cbc567c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:58.827798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a92fa0-1c8f-4002-a2e3-64d747528ce8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:58.829215 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a92fa0-1c8f-4002-a2e3-64d747528ce8 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-283312ff-7121-4662-bf62-3f184cbc567c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:58.839803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:58.840935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:58.841863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:58.841863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:58.841863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:58.841863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 557/1084] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:58.842101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce28fdf-6259-49bd-9f55-81a2d9f87bf8 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:58.842357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce28fdf-6259-49bd-9f55-81a2d9f87bf8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:58.842642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce28fdf-6259-49bd-9f55-81a2d9f87bf8 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:58.844430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 558/1085] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:27:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:27:58.846972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce28fdf-6259-49bd-9f55-81a2d9f87bf8 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:58.899576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:58.933790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 528/1086] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:27:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 24 03:27:58.935289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1512b5d-7c67-406e-b2e3-bef329ea88f9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:58.935578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1512b5d-7c67-406e-b2e3-bef329ea88f9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:58.935850 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1512b5d-7c67-406e-b2e3-bef329ea88f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:58.949015 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce28fdf-6259-49bd-9f55-81a2d9f87bf8 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:58.953534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-b1512b5d-7c67-406e-b2e3-bef329ea88f9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:27:58.954763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b1512b5d-7c67-406e-b2e3-bef329ea88f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:27:58.982687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1301f74f-b56a-4e0f-99ea-7923120bda06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:58.983293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1301f74f-b56a-4e0f-99ea-7923120bda06 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:58.983571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1301f74f-b56a-4e0f-99ea-7923120bda06 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:58.983842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1301f74f-b56a-4e0f-99ea-7923120bda06 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:58.984480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1301f74f-b56a-4e0f-99ea-7923120bda06 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:58.984780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1301f74f-b56a-4e0f-99ea-7923120bda06 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:58.986142 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1301f74f-b56a-4e0f-99ea-7923120bda06 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:58.997549 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 559/1087] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:58.998618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1301f74f-b56a-4e0f-99ea-7923120bda06 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:59.025619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 529/1088] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:27:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:27:59.062746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:59.120864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1301f74f-b56a-4e0f-99ea-7923120bda06 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:59.160852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e5b854-dc4c-418e-859a-d99ad4c9f41e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:59.161482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e5b854-dc4c-418e-859a-d99ad4c9f41e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:59.161755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e5b854-dc4c-418e-859a-d99ad4c9f41e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:59.162018 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e5b854-dc4c-418e-859a-d99ad4c9f41e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:59.172390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e5b854-dc4c-418e-859a-d99ad4c9f41e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:59.174389 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e5b854-dc4c-418e-859a-d99ad4c9f41e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-29e5b854-dc4c-418e-859a-d99ad4c9f41e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:27:59.190047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:59.191125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:27:59.192067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:27:59.192067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:27:59.192067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:59.192067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 560/1089] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:27:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:27:59.197208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1512b5d-7c67-406e-b2e3-bef329ea88f9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:59.293261 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e5b854-dc4c-418e-859a-d99ad4c9f41e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:59.306223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:59.310077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1512b5d-7c67-406e-b2e3-bef329ea88f9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:59.346433 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c79706c-567f-4252-9807-279a4f804625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:59.347032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c79706c-567f-4252-9807-279a4f804625 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:59.347309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c79706c-567f-4252-9807-279a4f804625 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:59.347578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c79706c-567f-4252-9807-279a4f804625 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:59.348229 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c79706c-567f-4252-9807-279a4f804625 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:59.348528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c79706c-567f-4252-9807-279a4f804625 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:59.349881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c79706c-567f-4252-9807-279a4f804625 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:59.360865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 530/1090] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:27:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:59.367089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c79706c-567f-4252-9807-279a4f804625 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:59.400721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:59.404579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e5b854-dc4c-418e-859a-d99ad4c9f41e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:59.438723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709902fa-147f-4124-8319-e017e3fa3315 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:59.440004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709902fa-147f-4124-8319-e017e3fa3315 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:59.440295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709902fa-147f-4124-8319-e017e3fa3315 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:59.440561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709902fa-147f-4124-8319-e017e3fa3315 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:59.450796 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709902fa-147f-4124-8319-e017e3fa3315 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-709902fa-147f-4124-8319-e017e3fa3315 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:27:59.455441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:59.456481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:27:59.457718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 561/1091] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:27:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:59.462547 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709902fa-147f-4124-8319-e017e3fa3315 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:59.473417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c79706c-567f-4252-9807-279a4f804625 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:59.507078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25d47c9-816b-4ce3-a5d2-a188b5fa2842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:59.507706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25d47c9-816b-4ce3-a5d2-a188b5fa2842 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:59.507985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25d47c9-816b-4ce3-a5d2-a188b5fa2842 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:59.508256 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25d47c9-816b-4ce3-a5d2-a188b5fa2842 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:59.514904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25d47c9-816b-4ce3-a5d2-a188b5fa2842 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:59.516348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25d47c9-816b-4ce3-a5d2-a188b5fa2842 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-f25d47c9-816b-4ce3-a5d2-a188b5fa2842 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:27:59.528140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:27:59.529060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:27:59.530428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:27:59.530428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:27:59.530428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:27:59.530428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:27:59.530428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:27:59.530428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:27:59.530428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:27:59.530428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:27:59.530428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 531/1092] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:27:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:27:59.535839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25d47c9-816b-4ce3-a5d2-a188b5fa2842 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:59.555471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:59.560969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709902fa-147f-4124-8319-e017e3fa3315 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:59.595465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8def3e5-beee-4e2a-9eca-95a420c28f50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:59.596281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8def3e5-beee-4e2a-9eca-95a420c28f50 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:59.596560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8def3e5-beee-4e2a-9eca-95a420c28f50 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:59.596831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8def3e5-beee-4e2a-9eca-95a420c28f50 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:59.597488 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8def3e5-beee-4e2a-9eca-95a420c28f50 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:59.597787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8def3e5-beee-4e2a-9eca-95a420c28f50 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:59.599175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8def3e5-beee-4e2a-9eca-95a420c28f50 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:59.607623 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 562/1093] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:27:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:59.613463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8def3e5-beee-4e2a-9eca-95a420c28f50 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:59.635103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:59.635473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25d47c9-816b-4ce3-a5d2-a188b5fa2842 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:59.668864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb549ed3-1a08-41a9-a125-10f71724f7fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:59.669463 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb549ed3-1a08-41a9-a125-10f71724f7fd None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:59.669741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb549ed3-1a08-41a9-a125-10f71724f7fd None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:59.670009 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb549ed3-1a08-41a9-a125-10f71724f7fd None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:59.670723 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb549ed3-1a08-41a9-a125-10f71724f7fd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:59.671020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb549ed3-1a08-41a9-a125-10f71724f7fd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:59.672436 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb549ed3-1a08-41a9-a125-10f71724f7fd None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:59.684631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 532/1094] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:27:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:59.690792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb549ed3-1a08-41a9-a125-10f71724f7fd None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:59.715027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8def3e5-beee-4e2a-9eca-95a420c28f50 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:59.749324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bb4740-0fed-4b34-b121-c5aef8f83358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:59.749941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bb4740-0fed-4b34-b121-c5aef8f83358 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:59.750243 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bb4740-0fed-4b34-b121-c5aef8f83358 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:59.750514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bb4740-0fed-4b34-b121-c5aef8f83358 None admin] PATH_INFO: `/v3/users/2cecc223594344cc9c0c1c2230d125e5` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:59.751639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41bb4740-0fed-4b34-b121-c5aef8f83358 None admin] RBAC: Authorizing `identity:get_user(user_id=2cecc223594344cc9c0c1c2230d125e5)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:59.753051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41bb4740-0fed-4b34-b121-c5aef8f83358 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:59.754778 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 563/1095] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:27:59 2022] GET /identity/v3/users/2cecc223594344cc9c0c1c2230d125e5 => generated 285 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:27:59.760492 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bb4740-0fed-4b34-b121-c5aef8f83358 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:27:59.767845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:59.788972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb549ed3-1a08-41a9-a125-10f71724f7fd None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:59.823369 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a6cb50-b9bb-4d99-b014-6c327d4bd0b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:59.823892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a6cb50-b9bb-4d99-b014-6c327d4bd0b9 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:59.824163 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a6cb50-b9bb-4d99-b014-6c327d4bd0b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:59.824427 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a6cb50-b9bb-4d99-b014-6c327d4bd0b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:59.825142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a6cb50-b9bb-4d99-b014-6c327d4bd0b9 None admin] RBAC: Adding query filter params (role.id=1489a00dd22146ecb6161ff7d549f95c, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=e5896018712d4b82a781365e478ee855) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:27:59.825440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a6cb50-b9bb-4d99-b014-6c327d4bd0b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:59.826906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a6cb50-b9bb-4d99-b014-6c327d4bd0b9 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:59.834880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 533/1096] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:27:59 2022] GET /identity/v3/role_assignments?user.id=e5896018712d4b82a781365e478ee855&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=1489a00dd22146ecb6161ff7d549f95c => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:59.848002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:27:59.861226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bb4740-0fed-4b34-b121-c5aef8f83358 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:27:59.900030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7937170c-e7d4-442c-b43d-906c6ee2f442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:27:59.900674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7937170c-e7d4-442c-b43d-906c6ee2f442 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:27:59.900955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7937170c-e7d4-442c-b43d-906c6ee2f442 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:27:59.901228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7937170c-e7d4-442c-b43d-906c6ee2f442 None admin] PATH_INFO: `/v3/projects/288ff8851fff49c1b5c75e576bee9d4b` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:27:59.902429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7937170c-e7d4-442c-b43d-906c6ee2f442 None admin] RBAC: Authorizing `identity:get_project(project_id=288ff8851fff49c1b5c75e576bee9d4b)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:27:59.903889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7937170c-e7d4-442c-b43d-906c6ee2f442 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:27:59.905565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 564/1097] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:27:59 2022] GET /identity/v3/projects/288ff8851fff49c1b5c75e576bee9d4b => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:27:59.911974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a6cb50-b9bb-4d99-b014-6c327d4bd0b9 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:00.015770 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a6cb50-b9bb-4d99-b014-6c327d4bd0b9 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:00.049823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23316303-6223-44e7-bcf5-3596583c3df7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:00.050508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23316303-6223-44e7-bcf5-3596583c3df7 None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:00.050787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23316303-6223-44e7-bcf5-3596583c3df7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:00.051059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23316303-6223-44e7-bcf5-3596583c3df7 None admin] PATH_INFO: `/v3/projects/288ff8851fff49c1b5c75e576bee9d4b/users/2cecc223594344cc9c0c1c2230d125e5/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:00.053010 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23316303-6223-44e7-bcf5-3596583c3df7 None admin] RBAC: Authorizing `identity:create_grant(project_id=288ff8851fff49c1b5c75e576bee9d4b, user_id=2cecc223594344cc9c0c1c2230d125e5, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:00.054477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23316303-6223-44e7-bcf5-3596583c3df7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:00.121989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 534/1098] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:27:59 2022] PUT /identity/v3/projects/288ff8851fff49c1b5c75e576bee9d4b/users/2cecc223594344cc9c0c1c2230d125e5/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:28:02.009069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a485d72a-b7d2-44d8-808b-f3d21af1d622 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:02.009597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a485d72a-b7d2-44d8-808b-f3d21af1d622 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:02.009597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a485d72a-b7d2-44d8-808b-f3d21af1d622 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:02.011009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 565/1099] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:02.016119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22138418-b9b0-4344-b7fe-058010a3f041 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:02.016486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22138418-b9b0-4344-b7fe-058010a3f041 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:02.016677 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22138418-b9b0-4344-b7fe-058010a3f041 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:02.023142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:02.035389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-22138418-b9b0-4344-b7fe-058010a3f041 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:02.036579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-22138418-b9b0-4344-b7fe-058010a3f041 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:02.101518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 535/1100] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:02.373187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a485d72a-b7d2-44d8-808b-f3d21af1d622 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:02.486513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a485d72a-b7d2-44d8-808b-f3d21af1d622 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:02.520248 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497b4426-6b8c-4d78-95c4-bcef6a6cb31a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:02.520861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497b4426-6b8c-4d78-95c4-bcef6a6cb31a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:02.521139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497b4426-6b8c-4d78-95c4-bcef6a6cb31a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:02.521411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497b4426-6b8c-4d78-95c4-bcef6a6cb31a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:02.531527 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497b4426-6b8c-4d78-95c4-bcef6a6cb31a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-497b4426-6b8c-4d78-95c4-bcef6a6cb31a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:28:02.536996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:28:02.538042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:28:02.539315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 566/1101] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:28:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:28:02.544258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22138418-b9b0-4344-b7fe-058010a3f041 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:02.640367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22138418-b9b0-4344-b7fe-058010a3f041 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:02.668781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:02.673420 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b3047-ef68-423b-b605-93fc9915d885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:02.674276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b3047-ef68-423b-b605-93fc9915d885 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:02.674612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b3047-ef68-423b-b605-93fc9915d885 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:02.674879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b3047-ef68-423b-b605-93fc9915d885 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:02.675529 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b3047-ef68-423b-b605-93fc9915d885 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:02.675825 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b3047-ef68-423b-b605-93fc9915d885 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:02.677199 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b3047-ef68-423b-b605-93fc9915d885 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:02.684893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 536/1102] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:28:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:02.691214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497b4426-6b8c-4d78-95c4-bcef6a6cb31a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:02.709906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:02.789477 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497b4426-6b8c-4d78-95c4-bcef6a6cb31a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:02.822073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cffb42-b544-4012-bbdd-d1d3f1465d2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:02.822716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cffb42-b544-4012-bbdd-d1d3f1465d2d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:02.822998 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cffb42-b544-4012-bbdd-d1d3f1465d2d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:02.823270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cffb42-b544-4012-bbdd-d1d3f1465d2d None admin] PATH_INFO: `/v3/users/2cecc223594344cc9c0c1c2230d125e5` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:02.824379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cffb42-b544-4012-bbdd-d1d3f1465d2d None admin] RBAC: Authorizing `identity:get_user(user_id=2cecc223594344cc9c0c1c2230d125e5)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:02.825802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cffb42-b544-4012-bbdd-d1d3f1465d2d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:02.827378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 567/1103] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:02 2022] GET /identity/v3/users/2cecc223594344cc9c0c1c2230d125e5 => generated 285 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:02.833794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b3047-ef68-423b-b605-93fc9915d885 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:02.931922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b3047-ef68-423b-b605-93fc9915d885 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:02.963782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba46533-4278-4873-8f2f-7315f9694c09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:02.964364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba46533-4278-4873-8f2f-7315f9694c09 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:02.964637 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba46533-4278-4873-8f2f-7315f9694c09 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:02.964902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba46533-4278-4873-8f2f-7315f9694c09 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:03.022193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:03.029134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba46533-4278-4873-8f2f-7315f9694c09 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:03.030581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba46533-4278-4873-8f2f-7315f9694c09 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:03.124468 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 537/1104] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:28:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:28:03.130989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cffb42-b544-4012-bbdd-d1d3f1465d2d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:03.212545 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:03.227749 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cffb42-b544-4012-bbdd-d1d3f1465d2d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:03.260500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1349e5-5b94-457c-9dbf-a30cf7c44e37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:03.261117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1349e5-5b94-457c-9dbf-a30cf7c44e37 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:03.261391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1349e5-5b94-457c-9dbf-a30cf7c44e37 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:03.261658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1349e5-5b94-457c-9dbf-a30cf7c44e37 None admin] PATH_INFO: `/v3/projects/288ff8851fff49c1b5c75e576bee9d4b` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:03.262756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1349e5-5b94-457c-9dbf-a30cf7c44e37 None admin] RBAC: Authorizing `identity:get_project(project_id=288ff8851fff49c1b5c75e576bee9d4b)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:03.264167 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1349e5-5b94-457c-9dbf-a30cf7c44e37 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:03.265740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 568/1105] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:28:03 2022] GET /identity/v3/projects/288ff8851fff49c1b5c75e576bee9d4b => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:03.272646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba46533-4278-4873-8f2f-7315f9694c09 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:03.362324 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:03.368498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba46533-4278-4873-8f2f-7315f9694c09 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:03.402131 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ecd9a3-e0af-4504-8542-9984da2db33d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:03.402747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ecd9a3-e0af-4504-8542-9984da2db33d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:03.403022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ecd9a3-e0af-4504-8542-9984da2db33d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:03.403290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ecd9a3-e0af-4504-8542-9984da2db33d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:03.404001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ecd9a3-e0af-4504-8542-9984da2db33d None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=288ff8851fff49c1b5c75e576bee9d4b, user.id=2cecc223594344cc9c0c1c2230d125e5) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:03.404302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ecd9a3-e0af-4504-8542-9984da2db33d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:03.405710 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ecd9a3-e0af-4504-8542-9984da2db33d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:03.411602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 538/1106] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:28:03 2022] GET /identity/v3/role_assignments?user.id=2cecc223594344cc9c0c1c2230d125e5&scope.project.id=288ff8851fff49c1b5c75e576bee9d4b&role.id=43ca25467bc6484da3a50508efb79230 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:05.328620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdff148-8d71-4429-9f7d-d43e27f06afe None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:05.329173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdff148-8d71-4429-9f7d-d43e27f06afe None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:05.329173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdff148-8d71-4429-9f7d-d43e27f06afe None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:05.331239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 569/1107] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:05.336528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1695582-0260-4dd4-ac17-304e82b3c1ef None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:05.336908 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1695582-0260-4dd4-ac17-304e82b3c1ef None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:05.337076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1695582-0260-4dd4-ac17-304e82b3c1ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:05.355371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d1695582-0260-4dd4-ac17-304e82b3c1ef None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:05.356584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d1695582-0260-4dd4-ac17-304e82b3c1ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:05.423587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 539/1108] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:28:05.692758 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdff148-8d71-4429-9f7d-d43e27f06afe None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:05.792118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdff148-8d71-4429-9f7d-d43e27f06afe None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:05.825229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51424703-d6e0-4fb2-a219-a7531603ded8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:05.825847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51424703-d6e0-4fb2-a219-a7531603ded8 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:05.826121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51424703-d6e0-4fb2-a219-a7531603ded8 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:05.826428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51424703-d6e0-4fb2-a219-a7531603ded8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:05.827526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51424703-d6e0-4fb2-a219-a7531603ded8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:05.828938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51424703-d6e0-4fb2-a219-a7531603ded8 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:05.831136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 570/1109] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:28:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:05.836767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1695582-0260-4dd4-ac17-304e82b3c1ef None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:05.920138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:05.931831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1695582-0260-4dd4-ac17-304e82b3c1ef None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:05.964500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5eba76-9b5b-4728-8ac8-1d776bebcc4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:05.965028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5eba76-9b5b-4728-8ac8-1d776bebcc4c None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:05.965303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5eba76-9b5b-4728-8ac8-1d776bebcc4c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:05.965569 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5eba76-9b5b-4728-8ac8-1d776bebcc4c None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:05.966222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5eba76-9b5b-4728-8ac8-1d776bebcc4c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:05.967624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5eba76-9b5b-4728-8ac8-1d776bebcc4c None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:06.046976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 540/1110] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:28:05 2022] POST /identity/v3/users => generated 286 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:28:07.982603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22b5259-07e6-4f17-9699-898189225d8c None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:07.983144 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22b5259-07e6-4f17-9699-898189225d8c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:07.983258 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22b5259-07e6-4f17-9699-898189225d8c None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:07.984504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 571/1111] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:07.990408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42d9aa6-9aa6-45b8-8a48-32af737b6001 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:07.990847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42d9aa6-9aa6-45b8-8a48-32af737b6001 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:07.990948 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42d9aa6-9aa6-45b8-8a48-32af737b6001 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:08.008838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f42d9aa6-9aa6-45b8-8a48-32af737b6001 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:08.010026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f42d9aa6-9aa6-45b8-8a48-32af737b6001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:08.074758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 541/1112] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:08.341931 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22b5259-07e6-4f17-9699-898189225d8c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:08.363835 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:08.441802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22b5259-07e6-4f17-9699-898189225d8c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:08.473984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f891b93f-53cf-456e-b683-57b1f2ef6a76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:08.474631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f891b93f-53cf-456e-b683-57b1f2ef6a76 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:08.474910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f891b93f-53cf-456e-b683-57b1f2ef6a76 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:08.475179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f891b93f-53cf-456e-b683-57b1f2ef6a76 None admin] PATH_INFO: `/v3/roles/d02a8e6c49b74aaa9403f8bd535b90bf` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:08.476557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f891b93f-53cf-456e-b683-57b1f2ef6a76 None admin] RBAC: Authorizing `identity:get_role(role_id=d02a8e6c49b74aaa9403f8bd535b90bf)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:08.478069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f891b93f-53cf-456e-b683-57b1f2ef6a76 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:08.479393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 572/1113] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:08 2022] GET /identity/v3/roles/d02a8e6c49b74aaa9403f8bd535b90bf => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:08.485187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42d9aa6-9aa6-45b8-8a48-32af737b6001 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:08.559264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:08.583061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42d9aa6-9aa6-45b8-8a48-32af737b6001 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:08.616034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e551b402-cdcd-4974-9c2e-9d2960b4d590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:08.616651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e551b402-cdcd-4974-9c2e-9d2960b4d590 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:08.616927 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e551b402-cdcd-4974-9c2e-9d2960b4d590 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:08.617195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e551b402-cdcd-4974-9c2e-9d2960b4d590 None admin] PATH_INFO: `/v3/users/fc07961de78c49ae8ecd9f6536d8f9ff` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:08.630328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e551b402-cdcd-4974-9c2e-9d2960b4d590 None admin] RBAC: Authorizing `identity:get_user(user_id=fc07961de78c49ae8ecd9f6536d8f9ff)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:08.632020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e551b402-cdcd-4974-9c2e-9d2960b4d590 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:08.633569 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 542/1114] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:08 2022] GET /identity/v3/users/fc07961de78c49ae8ecd9f6536d8f9ff => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:08.639280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f891b93f-53cf-456e-b683-57b1f2ef6a76 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:08.725500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:08.737687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f891b93f-53cf-456e-b683-57b1f2ef6a76 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:08.770638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f41e9f6-3f2b-4a66-b238-e0e5dc0e1291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:08.771235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f41e9f6-3f2b-4a66-b238-e0e5dc0e1291 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:08.771517 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f41e9f6-3f2b-4a66-b238-e0e5dc0e1291 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:08.771793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f41e9f6-3f2b-4a66-b238-e0e5dc0e1291 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:08.836082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f41e9f6-3f2b-4a66-b238-e0e5dc0e1291 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:08.837529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f41e9f6-3f2b-4a66-b238-e0e5dc0e1291 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:08.921238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:08.933122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 573/1115] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:28:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:28:08.939083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e551b402-cdcd-4974-9c2e-9d2960b4d590 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:09.036154 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e551b402-cdcd-4974-9c2e-9d2960b4d590 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:09.064006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:09.068859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17139870-6249-4cb4-a48b-71cef76595a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:09.069473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17139870-6249-4cb4-a48b-71cef76595a6 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:09.069752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17139870-6249-4cb4-a48b-71cef76595a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:09.070024 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17139870-6249-4cb4-a48b-71cef76595a6 None admin] PATH_INFO: `/v3/projects/288ff8851fff49c1b5c75e576bee9d4b` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:09.071159 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17139870-6249-4cb4-a48b-71cef76595a6 None admin] RBAC: Authorizing `identity:get_project(project_id=288ff8851fff49c1b5c75e576bee9d4b)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:09.072613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17139870-6249-4cb4-a48b-71cef76595a6 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:09.079804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 543/1116] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:28:08 2022] GET /identity/v3/projects/288ff8851fff49c1b5c75e576bee9d4b => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:09.080913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f41e9f6-3f2b-4a66-b238-e0e5dc0e1291 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:09.176690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f41e9f6-3f2b-4a66-b238-e0e5dc0e1291 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:09.208794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f117aa60-b0e5-47bd-9a1a-b8484fcd3e5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:09.209378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f117aa60-b0e5-47bd-9a1a-b8484fcd3e5e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:09.209650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f117aa60-b0e5-47bd-9a1a-b8484fcd3e5e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:09.209914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f117aa60-b0e5-47bd-9a1a-b8484fcd3e5e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:09.210658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f117aa60-b0e5-47bd-9a1a-b8484fcd3e5e None admin] RBAC: Adding query filter params (role.id=d02a8e6c49b74aaa9403f8bd535b90bf, scope.project.id=288ff8851fff49c1b5c75e576bee9d4b, user.id=fc07961de78c49ae8ecd9f6536d8f9ff) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:09.210958 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f117aa60-b0e5-47bd-9a1a-b8484fcd3e5e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:09.212360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f117aa60-b0e5-47bd-9a1a-b8484fcd3e5e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:09.217462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 574/1117] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:28:09 2022] GET /identity/v3/role_assignments?user.id=fc07961de78c49ae8ecd9f6536d8f9ff&scope.project.id=288ff8851fff49c1b5c75e576bee9d4b&role.id=d02a8e6c49b74aaa9403f8bd535b90bf => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:11.069890 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8fd33a-8e20-42bc-9921-2ad8b938797e None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:11.070473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8fd33a-8e20-42bc-9921-2ad8b938797e None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:11.070610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8fd33a-8e20-42bc-9921-2ad8b938797e None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:11.071947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 544/1118] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:11.077161 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81d3c5f-52e2-4598-9a01-c38008da918a None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:11.077596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81d3c5f-52e2-4598-9a01-c38008da918a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:11.077696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81d3c5f-52e2-4598-9a01-c38008da918a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:11.096448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d81d3c5f-52e2-4598-9a01-c38008da918a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:11.097622 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d81d3c5f-52e2-4598-9a01-c38008da918a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:11.143819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:11.162430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 575/1119] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:11.437924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8fd33a-8e20-42bc-9921-2ad8b938797e None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:11.537326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8fd33a-8e20-42bc-9921-2ad8b938797e None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:11.569881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e544f7-8f3d-4cb5-9121-fc7468074412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:11.570519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e544f7-8f3d-4cb5-9121-fc7468074412 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:11.570792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e544f7-8f3d-4cb5-9121-fc7468074412 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:11.571060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e544f7-8f3d-4cb5-9121-fc7468074412 None admin] PATH_INFO: `/v3/roles/d02a8e6c49b74aaa9403f8bd535b90bf` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:11.572437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e544f7-8f3d-4cb5-9121-fc7468074412 None admin] RBAC: Authorizing `identity:get_role(role_id=d02a8e6c49b74aaa9403f8bd535b90bf)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:11.573788 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e544f7-8f3d-4cb5-9121-fc7468074412 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:11.575102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 545/1120] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:11 2022] GET /identity/v3/roles/d02a8e6c49b74aaa9403f8bd535b90bf => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:11.581006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81d3c5f-52e2-4598-9a01-c38008da918a None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:11.678661 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81d3c5f-52e2-4598-9a01-c38008da918a None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:11.711324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5c8463-573e-4b5f-af20-d3aa3b11015e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:11.711941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5c8463-573e-4b5f-af20-d3aa3b11015e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:11.712221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5c8463-573e-4b5f-af20-d3aa3b11015e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:11.712495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5c8463-573e-4b5f-af20-d3aa3b11015e None admin] PATH_INFO: `/v3/users/fc07961de78c49ae8ecd9f6536d8f9ff` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:11.713619 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5c8463-573e-4b5f-af20-d3aa3b11015e None admin] RBAC: Authorizing `identity:get_user(user_id=fc07961de78c49ae8ecd9f6536d8f9ff)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:11.715069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5c8463-573e-4b5f-af20-d3aa3b11015e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:11.716610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 576/1121] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:11 2022] GET /identity/v3/users/fc07961de78c49ae8ecd9f6536d8f9ff => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:11.723178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e544f7-8f3d-4cb5-9121-fc7468074412 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:11.787779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:11.819377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e544f7-8f3d-4cb5-9121-fc7468074412 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:11.852304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae25ac4-ac33-407c-aeaf-63a9e996a88e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:11.852914 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae25ac4-ac33-407c-aeaf-63a9e996a88e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:11.853190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae25ac4-ac33-407c-aeaf-63a9e996a88e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:11.853461 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae25ac4-ac33-407c-aeaf-63a9e996a88e None admin] PATH_INFO: `/v3/projects/288ff8851fff49c1b5c75e576bee9d4b` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:11.854588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae25ac4-ac33-407c-aeaf-63a9e996a88e None admin] RBAC: Authorizing `identity:get_project(project_id=288ff8851fff49c1b5c75e576bee9d4b)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:11.856003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae25ac4-ac33-407c-aeaf-63a9e996a88e None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:11.857591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 546/1122] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:28:11 2022] GET /identity/v3/projects/288ff8851fff49c1b5c75e576bee9d4b => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:11.864252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5c8463-573e-4b5f-af20-d3aa3b11015e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:11.913432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:11.960596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5c8463-573e-4b5f-af20-d3aa3b11015e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:11.993682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1832abfc-640d-426c-a241-97e17a850e0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:11.994351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1832abfc-640d-426c-a241-97e17a850e0d None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:11.994634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1832abfc-640d-426c-a241-97e17a850e0d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:11.994905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1832abfc-640d-426c-a241-97e17a850e0d None admin] PATH_INFO: `/v3/projects/288ff8851fff49c1b5c75e576bee9d4b/users/fc07961de78c49ae8ecd9f6536d8f9ff/roles/d02a8e6c49b74aaa9403f8bd535b90bf` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:11.997040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1832abfc-640d-426c-a241-97e17a850e0d None admin] RBAC: Authorizing `identity:create_grant(project_id=288ff8851fff49c1b5c75e576bee9d4b, user_id=fc07961de78c49ae8ecd9f6536d8f9ff, role_id=d02a8e6c49b74aaa9403f8bd535b90bf)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:11.998491 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1832abfc-640d-426c-a241-97e17a850e0d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:12.064270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 577/1123] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:28:11 2022] PUT /identity/v3/projects/288ff8851fff49c1b5c75e576bee9d4b/users/fc07961de78c49ae8ecd9f6536d8f9ff/roles/d02a8e6c49b74aaa9403f8bd535b90bf => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:28:13.943735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a48d4a-c6b4-45a9-9e61-09171fe65f6a None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:13.944286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a48d4a-c6b4-45a9-9e61-09171fe65f6a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:13.944286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a48d4a-c6b4-45a9-9e61-09171fe65f6a None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:13.945612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 547/1124] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:13.951034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0672f3-bdab-40f0-9433-3eed047b03b3 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:13.951421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0672f3-bdab-40f0-9433-3eed047b03b3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:13.951578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0672f3-bdab-40f0-9433-3eed047b03b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:13.969864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-fb0672f3-bdab-40f0-9433-3eed047b03b3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:13.971955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-fb0672f3-bdab-40f0-9433-3eed047b03b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:14.045135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 578/1125] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:14.355830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a48d4a-c6b4-45a9-9e61-09171fe65f6a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:14.465285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a48d4a-c6b4-45a9-9e61-09171fe65f6a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:14.499721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11101e44-61d3-466a-b56a-4d82eca243fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:14.500354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11101e44-61d3-466a-b56a-4d82eca243fb None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:14.500634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11101e44-61d3-466a-b56a-4d82eca243fb None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:14.500906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11101e44-61d3-466a-b56a-4d82eca243fb None admin] PATH_INFO: `/v3/roles/d02a8e6c49b74aaa9403f8bd535b90bf` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:14.503278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11101e44-61d3-466a-b56a-4d82eca243fb None admin] RBAC: Authorizing `identity:get_role(role_id=d02a8e6c49b74aaa9403f8bd535b90bf)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:14.504674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11101e44-61d3-466a-b56a-4d82eca243fb None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:14.506291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 548/1126] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:14 2022] GET /identity/v3/roles/d02a8e6c49b74aaa9403f8bd535b90bf => generated 223 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:14.512339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0672f3-bdab-40f0-9433-3eed047b03b3 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:14.557729 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:14.620089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0672f3-bdab-40f0-9433-3eed047b03b3 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:14.657044 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2454fa8e-7ba7-45d1-b664-765ac70f1c08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:14.657803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2454fa8e-7ba7-45d1-b664-765ac70f1c08 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:14.658853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2454fa8e-7ba7-45d1-b664-765ac70f1c08 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:14.659352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2454fa8e-7ba7-45d1-b664-765ac70f1c08 None admin] PATH_INFO: `/v3/users/fc07961de78c49ae8ecd9f6536d8f9ff` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:14.660706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2454fa8e-7ba7-45d1-b664-765ac70f1c08 None admin] RBAC: Authorizing `identity:get_user(user_id=fc07961de78c49ae8ecd9f6536d8f9ff)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:14.662706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2454fa8e-7ba7-45d1-b664-765ac70f1c08 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:14.664287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 579/1127] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:14 2022] GET /identity/v3/users/fc07961de78c49ae8ecd9f6536d8f9ff => generated 286 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:14.670180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11101e44-61d3-466a-b56a-4d82eca243fb None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:14.674624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:14.776437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11101e44-61d3-466a-b56a-4d82eca243fb None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:14.809646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3a946e-cffb-4503-b111-bf10b275d1bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:14.810271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3a946e-cffb-4503-b111-bf10b275d1bd None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:14.810546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3a946e-cffb-4503-b111-bf10b275d1bd None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:14.810818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3a946e-cffb-4503-b111-bf10b275d1bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:14.875207 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3a946e-cffb-4503-b111-bf10b275d1bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:14.876569 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3a946e-cffb-4503-b111-bf10b275d1bd None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:14.879136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:14.979938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 549/1128] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:28:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:28:14.986052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2454fa8e-7ba7-45d1-b664-765ac70f1c08 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:15.084163 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2454fa8e-7ba7-45d1-b664-765ac70f1c08 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:15.092976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:15.120203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5311ece0-85c4-4897-90a8-210dae3df26d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:15.121269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5311ece0-85c4-4897-90a8-210dae3df26d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:15.121775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5311ece0-85c4-4897-90a8-210dae3df26d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:15.122311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5311ece0-85c4-4897-90a8-210dae3df26d None admin] PATH_INFO: `/v3/projects/288ff8851fff49c1b5c75e576bee9d4b` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:15.124041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5311ece0-85c4-4897-90a8-210dae3df26d None admin] RBAC: Authorizing `identity:get_project(project_id=288ff8851fff49c1b5c75e576bee9d4b)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:15.125525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5311ece0-85c4-4897-90a8-210dae3df26d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:15.127975 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 580/1129] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:28:14 2022] GET /identity/v3/projects/288ff8851fff49c1b5c75e576bee9d4b => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:15.134364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3a946e-cffb-4503-b111-bf10b275d1bd None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:15.229757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3a946e-cffb-4503-b111-bf10b275d1bd None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:15.232042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:15.264459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e6d6aa-44be-4ae4-8dce-a23cad381571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:15.265070 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e6d6aa-44be-4ae4-8dce-a23cad381571 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:15.265539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e6d6aa-44be-4ae4-8dce-a23cad381571 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:15.266105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e6d6aa-44be-4ae4-8dce-a23cad381571 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:15.266863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e6d6aa-44be-4ae4-8dce-a23cad381571 None admin] RBAC: Adding query filter params (role.id=d02a8e6c49b74aaa9403f8bd535b90bf, scope.project.id=288ff8851fff49c1b5c75e576bee9d4b, user.id=fc07961de78c49ae8ecd9f6536d8f9ff) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:15.267165 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e6d6aa-44be-4ae4-8dce-a23cad381571 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:15.268968 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e6d6aa-44be-4ae4-8dce-a23cad381571 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:15.274576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 550/1130] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:28:15 2022] GET /identity/v3/role_assignments?user.id=fc07961de78c49ae8ecd9f6536d8f9ff&scope.project.id=288ff8851fff49c1b5c75e576bee9d4b&role.id=d02a8e6c49b74aaa9403f8bd535b90bf => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:17.171007 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78507dea-fe52-4cc7-aaee-e0c270b88206 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:17.171571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78507dea-fe52-4cc7-aaee-e0c270b88206 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:17.171571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78507dea-fe52-4cc7-aaee-e0c270b88206 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:17.172857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 581/1131] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:17.178812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a62e594-8597-452f-abe0-165cce4347d6 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:17.179187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a62e594-8597-452f-abe0-165cce4347d6 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:17.179353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a62e594-8597-452f-abe0-165cce4347d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:17.197136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-4a62e594-8597-452f-abe0-165cce4347d6 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:17.198355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4a62e594-8597-452f-abe0-165cce4347d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:17.262424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 551/1132] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:17.528046 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78507dea-fe52-4cc7-aaee-e0c270b88206 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:17.627972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78507dea-fe52-4cc7-aaee-e0c270b88206 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:17.660855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d4d283-4673-4d4e-b3d2-f134a15374f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:17.661461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d4d283-4673-4d4e-b3d2-f134a15374f2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:17.661736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d4d283-4673-4d4e-b3d2-f134a15374f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:17.662004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d4d283-4673-4d4e-b3d2-f134a15374f2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:17.663125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d4d283-4673-4d4e-b3d2-f134a15374f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:17.664536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d4d283-4673-4d4e-b3d2-f134a15374f2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:17.666054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 582/1133] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:28:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:17.671765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a62e594-8597-452f-abe0-165cce4347d6 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:17.764452 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:17.768189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a62e594-8597-452f-abe0-165cce4347d6 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:17.801717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601f04d9-db27-4f16-9394-1c103bac1ec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:17.802276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601f04d9-db27-4f16-9394-1c103bac1ec3 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:17.802551 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601f04d9-db27-4f16-9394-1c103bac1ec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:17.802816 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601f04d9-db27-4f16-9394-1c103bac1ec3 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:17.803439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601f04d9-db27-4f16-9394-1c103bac1ec3 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:17.804842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601f04d9-db27-4f16-9394-1c103bac1ec3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:17.888931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 552/1134] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:28:17 2022] POST /identity/v3/projects => generated 311 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:28:19.850819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9351e9e9-d724-4f41-936b-f7e7a062b60e None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:19.851352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9351e9e9-d724-4f41-936b-f7e7a062b60e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:19.851352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9351e9e9-d724-4f41-936b-f7e7a062b60e None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:19.852694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 583/1135] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:19.858088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d793b402-5e60-44dd-bb1e-2f8b90a4b352 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:19.858542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d793b402-5e60-44dd-bb1e-2f8b90a4b352 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:19.858679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d793b402-5e60-44dd-bb1e-2f8b90a4b352 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:19.877126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d793b402-5e60-44dd-bb1e-2f8b90a4b352 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:19.878359 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d793b402-5e60-44dd-bb1e-2f8b90a4b352 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:19.944229 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 553/1136] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:20.217802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9351e9e9-d724-4f41-936b-f7e7a062b60e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:20.265322 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:20.321351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9351e9e9-d724-4f41-936b-f7e7a062b60e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:20.355732 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23494d6b-26c6-4b74-bc37-163a2b45da20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:20.356356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23494d6b-26c6-4b74-bc37-163a2b45da20 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:20.357827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23494d6b-26c6-4b74-bc37-163a2b45da20 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:20.357827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23494d6b-26c6-4b74-bc37-163a2b45da20 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:20.358013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23494d6b-26c6-4b74-bc37-163a2b45da20 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:20.359451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23494d6b-26c6-4b74-bc37-163a2b45da20 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:20.361252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 584/1137] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:28:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:20.366846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d793b402-5e60-44dd-bb1e-2f8b90a4b352 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:20.445927 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:20.464612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d793b402-5e60-44dd-bb1e-2f8b90a4b352 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:20.497491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2deb708d-185f-4a2d-ac18-b888c2525fd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:20.498053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deb708d-185f-4a2d-ac18-b888c2525fd2 None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:20.498347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deb708d-185f-4a2d-ac18-b888c2525fd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:20.498611 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2deb708d-185f-4a2d-ac18-b888c2525fd2 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:20.499244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2deb708d-185f-4a2d-ac18-b888c2525fd2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:20.500654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2deb708d-185f-4a2d-ac18-b888c2525fd2 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:20.572793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 554/1138] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:28:20 2022] POST /identity/v3/users => generated 286 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:28:22.531316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42a3645-e310-4d2f-ae39-baef3c2a69d0 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:22.531858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42a3645-e310-4d2f-ae39-baef3c2a69d0 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:22.531858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42a3645-e310-4d2f-ae39-baef3c2a69d0 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:22.533799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 585/1139] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:22.539016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1f8bd8-40da-4225-aaa6-2bfa184e6a9a None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:22.539441 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1f8bd8-40da-4225-aaa6-2bfa184e6a9a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:22.539553 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1f8bd8-40da-4225-aaa6-2bfa184e6a9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:22.557239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-8f1f8bd8-40da-4225-aaa6-2bfa184e6a9a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:22.558443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-8f1f8bd8-40da-4225-aaa6-2bfa184e6a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:22.622061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 555/1140] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:22.890613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42a3645-e310-4d2f-ae39-baef3c2a69d0 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:22.990755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42a3645-e310-4d2f-ae39-baef3c2a69d0 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:23.003554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:23.023880 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fda8b0-ce18-4b87-9339-1fc9f7a67bb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:23.024491 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fda8b0-ce18-4b87-9339-1fc9f7a67bb9 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:23.024763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fda8b0-ce18-4b87-9339-1fc9f7a67bb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:23.025030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fda8b0-ce18-4b87-9339-1fc9f7a67bb9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:23.035447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fda8b0-ce18-4b87-9339-1fc9f7a67bb9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-d9fda8b0-ce18-4b87-9339-1fc9f7a67bb9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:28:23.040063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:28:23.041307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:28:23.042579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 586/1141] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:28:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:28:23.047141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1f8bd8-40da-4225-aaa6-2bfa184e6a9a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:23.110069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:23.144671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1f8bd8-40da-4225-aaa6-2bfa184e6a9a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:23.178141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423589de-fda2-40dc-bebc-e153c35c68f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:23.178756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423589de-fda2-40dc-bebc-e153c35c68f0 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:23.179027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423589de-fda2-40dc-bebc-e153c35c68f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:23.179292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423589de-fda2-40dc-bebc-e153c35c68f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:23.179943 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423589de-fda2-40dc-bebc-e153c35c68f0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:23.180238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423589de-fda2-40dc-bebc-e153c35c68f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:23.181576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423589de-fda2-40dc-bebc-e153c35c68f0 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:23.190738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 556/1142] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:28:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:23.196503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fda8b0-ce18-4b87-9339-1fc9f7a67bb9 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:23.293577 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fda8b0-ce18-4b87-9339-1fc9f7a67bb9 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:23.327526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30127230-3d77-4ad1-a2e9-1409c13460ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:23.328137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30127230-3d77-4ad1-a2e9-1409c13460ed None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:23.328421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30127230-3d77-4ad1-a2e9-1409c13460ed None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:23.328690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30127230-3d77-4ad1-a2e9-1409c13460ed None admin] PATH_INFO: `/v3/users/8341cfcf449e448bbdac5f84de4f9fec` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:23.341384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30127230-3d77-4ad1-a2e9-1409c13460ed None admin] RBAC: Authorizing `identity:get_user(user_id=8341cfcf449e448bbdac5f84de4f9fec)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:23.342911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30127230-3d77-4ad1-a2e9-1409c13460ed None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:23.344513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 587/1143] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:23 2022] GET /identity/v3/users/8341cfcf449e448bbdac5f84de4f9fec => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:23.350470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423589de-fda2-40dc-bebc-e153c35c68f0 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:23.448216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423589de-fda2-40dc-bebc-e153c35c68f0 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:23.473398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:23.480980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21c9a7b-fa48-485d-9d56-1245bcca3cfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:23.481570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21c9a7b-fa48-485d-9d56-1245bcca3cfe None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:23.481842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21c9a7b-fa48-485d-9d56-1245bcca3cfe None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:23.482106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21c9a7b-fa48-485d-9d56-1245bcca3cfe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:23.548145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21c9a7b-fa48-485d-9d56-1245bcca3cfe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:23.550506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21c9a7b-fa48-485d-9d56-1245bcca3cfe None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:23.645576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 557/1144] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:28:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 24 03:28:23.651501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30127230-3d77-4ad1-a2e9-1409c13460ed None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:23.680496 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:23.747839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30127230-3d77-4ad1-a2e9-1409c13460ed None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:23.782365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c465be9-aab7-489b-bb14-5ec9fc28c927 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:23.783050 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c465be9-aab7-489b-bb14-5ec9fc28c927 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:23.783556 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c465be9-aab7-489b-bb14-5ec9fc28c927 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:23.784005 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c465be9-aab7-489b-bb14-5ec9fc28c927 None admin] PATH_INFO: `/v3/projects/23bfa77e3bef4ab08a8435d9b37b89a7` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:23.785437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c465be9-aab7-489b-bb14-5ec9fc28c927 None admin] RBAC: Authorizing `identity:get_project(project_id=23bfa77e3bef4ab08a8435d9b37b89a7)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:23.786881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c465be9-aab7-489b-bb14-5ec9fc28c927 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:23.788473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 588/1145] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:28:23 2022] GET /identity/v3/projects/23bfa77e3bef4ab08a8435d9b37b89a7 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:23.794783 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21c9a7b-fa48-485d-9d56-1245bcca3cfe None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:23.821901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:23.891910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21c9a7b-fa48-485d-9d56-1245bcca3cfe None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:23.925581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc74c62-a556-4034-bce9-6ac5e60b9fdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:23.926194 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc74c62-a556-4034-bce9-6ac5e60b9fdb None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:23.926473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc74c62-a556-4034-bce9-6ac5e60b9fdb None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:23.926744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc74c62-a556-4034-bce9-6ac5e60b9fdb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:23.927466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc74c62-a556-4034-bce9-6ac5e60b9fdb None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=23bfa77e3bef4ab08a8435d9b37b89a7, user.id=8341cfcf449e448bbdac5f84de4f9fec) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:23.927771 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc74c62-a556-4034-bce9-6ac5e60b9fdb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:23.929188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc74c62-a556-4034-bce9-6ac5e60b9fdb None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:23.934458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 558/1146] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:28:23 2022] GET /identity/v3/role_assignments?user.id=8341cfcf449e448bbdac5f84de4f9fec&scope.project.id=23bfa77e3bef4ab08a8435d9b37b89a7&role.id=43ca25467bc6484da3a50508efb79230 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:25.819819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dd0d52-20d1-424b-a317-5ae39a6ee902 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:25.820332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dd0d52-20d1-424b-a317-5ae39a6ee902 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:25.820332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dd0d52-20d1-424b-a317-5ae39a6ee902 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:25.821614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 589/1147] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:25.826848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6f3e66-f27e-4e79-97b2-dfac7905f7ab None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:25.827257 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6f3e66-f27e-4e79-97b2-dfac7905f7ab None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:25.827433 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6f3e66-f27e-4e79-97b2-dfac7905f7ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:25.845484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1b6f3e66-f27e-4e79-97b2-dfac7905f7ab None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:25.846697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1b6f3e66-f27e-4e79-97b2-dfac7905f7ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:25.911414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 559/1148] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:26.183199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dd0d52-20d1-424b-a317-5ae39a6ee902 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:26.278056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:26.282428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dd0d52-20d1-424b-a317-5ae39a6ee902 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:26.315611 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca44c5b-8022-4b3b-9b44-0e56c99ca6e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:26.316224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca44c5b-8022-4b3b-9b44-0e56c99ca6e4 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:26.316504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca44c5b-8022-4b3b-9b44-0e56c99ca6e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:26.316777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca44c5b-8022-4b3b-9b44-0e56c99ca6e4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:26.326469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca44c5b-8022-4b3b-9b44-0e56c99ca6e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-cca44c5b-8022-4b3b-9b44-0e56c99ca6e4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:28:26.331782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:28:26.332889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:28:26.333738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 590/1149] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:28:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:28:26.338887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6f3e66-f27e-4e79-97b2-dfac7905f7ab None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:26.362233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:26.435895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6f3e66-f27e-4e79-97b2-dfac7905f7ab None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:26.468928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7278f0e3-38dd-4578-aa5a-c8796ca2418a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:26.469522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7278f0e3-38dd-4578-aa5a-c8796ca2418a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:26.469800 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7278f0e3-38dd-4578-aa5a-c8796ca2418a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:26.470071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7278f0e3-38dd-4578-aa5a-c8796ca2418a None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:26.471419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7278f0e3-38dd-4578-aa5a-c8796ca2418a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:26.471717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7278f0e3-38dd-4578-aa5a-c8796ca2418a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:26.473069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7278f0e3-38dd-4578-aa5a-c8796ca2418a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:26.481041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 560/1150] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:28:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:26.487203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca44c5b-8022-4b3b-9b44-0e56c99ca6e4 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:26.586084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca44c5b-8022-4b3b-9b44-0e56c99ca6e4 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:26.617984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b5b3d6-34fc-4b35-9ff6-b63e84a8b7fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:26.618628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b5b3d6-34fc-4b35-9ff6-b63e84a8b7fc None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:26.618909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b5b3d6-34fc-4b35-9ff6-b63e84a8b7fc None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:26.619181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b5b3d6-34fc-4b35-9ff6-b63e84a8b7fc None admin] PATH_INFO: `/v3/users/8341cfcf449e448bbdac5f84de4f9fec` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:26.620286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b5b3d6-34fc-4b35-9ff6-b63e84a8b7fc None admin] RBAC: Authorizing `identity:get_user(user_id=8341cfcf449e448bbdac5f84de4f9fec)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:26.621712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b5b3d6-34fc-4b35-9ff6-b63e84a8b7fc None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:26.623296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 591/1151] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:26 2022] GET /identity/v3/users/8341cfcf449e448bbdac5f84de4f9fec => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:26.629898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7278f0e3-38dd-4578-aa5a-c8796ca2418a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:26.711516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:26.727049 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7278f0e3-38dd-4578-aa5a-c8796ca2418a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:26.761962 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996f3c7a-5fdf-48be-984d-0f6a0d8db716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:26.762603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996f3c7a-5fdf-48be-984d-0f6a0d8db716 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:26.762878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996f3c7a-5fdf-48be-984d-0f6a0d8db716 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:26.763145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996f3c7a-5fdf-48be-984d-0f6a0d8db716 None admin] PATH_INFO: `/v3/projects/23bfa77e3bef4ab08a8435d9b37b89a7` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:26.764232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996f3c7a-5fdf-48be-984d-0f6a0d8db716 None admin] RBAC: Authorizing `identity:get_project(project_id=23bfa77e3bef4ab08a8435d9b37b89a7)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:26.765638 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996f3c7a-5fdf-48be-984d-0f6a0d8db716 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:26.767304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 561/1152] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:28:26 2022] GET /identity/v3/projects/23bfa77e3bef4ab08a8435d9b37b89a7 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:26.773455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b5b3d6-34fc-4b35-9ff6-b63e84a8b7fc None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:26.801489 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:26.870565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b5b3d6-34fc-4b35-9ff6-b63e84a8b7fc None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:26.903492 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1170d105-d47b-4cd4-95a1-ecfb2cdcf39b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:26.904145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1170d105-d47b-4cd4-95a1-ecfb2cdcf39b None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:26.904425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1170d105-d47b-4cd4-95a1-ecfb2cdcf39b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:26.904699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1170d105-d47b-4cd4-95a1-ecfb2cdcf39b None admin] PATH_INFO: `/v3/projects/23bfa77e3bef4ab08a8435d9b37b89a7/users/8341cfcf449e448bbdac5f84de4f9fec/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:26.906620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1170d105-d47b-4cd4-95a1-ecfb2cdcf39b None admin] RBAC: Authorizing `identity:create_grant(project_id=23bfa77e3bef4ab08a8435d9b37b89a7, user_id=8341cfcf449e448bbdac5f84de4f9fec, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:26.908655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1170d105-d47b-4cd4-95a1-ecfb2cdcf39b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:26.996834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 592/1153] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:28:26 2022] PUT /identity/v3/projects/23bfa77e3bef4ab08a8435d9b37b89a7/users/8341cfcf449e448bbdac5f84de4f9fec/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:28:28.874358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44395c66-8ace-43d0-beae-ea2a16b6b372 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:28.874897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44395c66-8ace-43d0-beae-ea2a16b6b372 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:28.874897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44395c66-8ace-43d0-beae-ea2a16b6b372 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:28.876228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 562/1154] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:28.882190 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a221f9e5-c38c-44ac-97a5-976ab6aff5f8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:28.882623 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a221f9e5-c38c-44ac-97a5-976ab6aff5f8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:28.882716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a221f9e5-c38c-44ac-97a5-976ab6aff5f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:28.901291 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-a221f9e5-c38c-44ac-97a5-976ab6aff5f8 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:28.902531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a221f9e5-c38c-44ac-97a5-976ab6aff5f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:28.969033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 593/1155] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:28:29.247547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44395c66-8ace-43d0-beae-ea2a16b6b372 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:29.347950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44395c66-8ace-43d0-beae-ea2a16b6b372 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:29.382262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4854e48-fe5f-4d84-ac3b-ff054136b832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:29.382881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4854e48-fe5f-4d84-ac3b-ff054136b832 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:29.383162 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4854e48-fe5f-4d84-ac3b-ff054136b832 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:29.383477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4854e48-fe5f-4d84-ac3b-ff054136b832 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:29.393352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4854e48-fe5f-4d84-ac3b-ff054136b832 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-a4854e48-fe5f-4d84-ac3b-ff054136b832 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:28:29.398080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:28:29.399221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:28:29.400432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 563/1156] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:28:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:28:29.405215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a221f9e5-c38c-44ac-97a5-976ab6aff5f8 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:29.424182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:29.506340 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a221f9e5-c38c-44ac-97a5-976ab6aff5f8 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:29.540924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72d2f02-3819-48e7-9a9e-f6974bb81503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:29.541524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72d2f02-3819-48e7-9a9e-f6974bb81503 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:29.541800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72d2f02-3819-48e7-9a9e-f6974bb81503 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:29.542070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72d2f02-3819-48e7-9a9e-f6974bb81503 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:29.542759 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72d2f02-3819-48e7-9a9e-f6974bb81503 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:29.543059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72d2f02-3819-48e7-9a9e-f6974bb81503 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:29.544412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72d2f02-3819-48e7-9a9e-f6974bb81503 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:29.553056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 594/1157] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:28:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:29.558923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4854e48-fe5f-4d84-ac3b-ff054136b832 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:29.574662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:29.659255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4854e48-fe5f-4d84-ac3b-ff054136b832 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:29.696609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a1b9a7-3d06-428c-ac1e-71776f65e883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:29.697220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a1b9a7-3d06-428c-ac1e-71776f65e883 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:29.697493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a1b9a7-3d06-428c-ac1e-71776f65e883 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:29.697758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a1b9a7-3d06-428c-ac1e-71776f65e883 None admin] PATH_INFO: `/v3/users/8341cfcf449e448bbdac5f84de4f9fec` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:29.698887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a1b9a7-3d06-428c-ac1e-71776f65e883 None admin] RBAC: Authorizing `identity:get_user(user_id=8341cfcf449e448bbdac5f84de4f9fec)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:29.700296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a1b9a7-3d06-428c-ac1e-71776f65e883 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:29.701841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 564/1158] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:29 2022] GET /identity/v3/users/8341cfcf449e448bbdac5f84de4f9fec => generated 286 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:29.708672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72d2f02-3819-48e7-9a9e-f6974bb81503 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:29.785909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:29.806866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72d2f02-3819-48e7-9a9e-f6974bb81503 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:29.840968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c16e62-f0d7-42d4-ab35-31bf78b9dc54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:29.841566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c16e62-f0d7-42d4-ab35-31bf78b9dc54 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:29.841847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c16e62-f0d7-42d4-ab35-31bf78b9dc54 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:29.842119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c16e62-f0d7-42d4-ab35-31bf78b9dc54 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:29.908362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c16e62-f0d7-42d4-ab35-31bf78b9dc54 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:29.909751 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c16e62-f0d7-42d4-ab35-31bf78b9dc54 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:29.993406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:30.015659 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 595/1159] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:28:29 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:28:30.021643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a1b9a7-3d06-428c-ac1e-71776f65e883 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:30.100028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:30.119177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a1b9a7-3d06-428c-ac1e-71776f65e883 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:30.154265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca842b4-c684-4568-8558-ef4a149a277d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:30.154892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca842b4-c684-4568-8558-ef4a149a277d None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:30.155159 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca842b4-c684-4568-8558-ef4a149a277d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:30.155426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca842b4-c684-4568-8558-ef4a149a277d None admin] PATH_INFO: `/v3/projects/23bfa77e3bef4ab08a8435d9b37b89a7` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:30.156532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca842b4-c684-4568-8558-ef4a149a277d None admin] RBAC: Authorizing `identity:get_project(project_id=23bfa77e3bef4ab08a8435d9b37b89a7)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:30.157995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca842b4-c684-4568-8558-ef4a149a277d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:30.159634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 565/1160] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:28:30 2022] GET /identity/v3/projects/23bfa77e3bef4ab08a8435d9b37b89a7 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:30.166135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c16e62-f0d7-42d4-ab35-31bf78b9dc54 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:30.265206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c16e62-f0d7-42d4-ab35-31bf78b9dc54 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:30.299260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9610d6-6cb4-4494-9544-12b2be977315 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:30.299852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9610d6-6cb4-4494-9544-12b2be977315 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:30.300133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9610d6-6cb4-4494-9544-12b2be977315 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:30.300400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9610d6-6cb4-4494-9544-12b2be977315 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:30.301135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9610d6-6cb4-4494-9544-12b2be977315 None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=23bfa77e3bef4ab08a8435d9b37b89a7, user.id=8341cfcf449e448bbdac5f84de4f9fec) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:30.301440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9610d6-6cb4-4494-9544-12b2be977315 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:30.302895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9610d6-6cb4-4494-9544-12b2be977315 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:30.309270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 596/1161] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:28:30 2022] GET /identity/v3/role_assignments?user.id=8341cfcf449e448bbdac5f84de4f9fec&scope.project.id=23bfa77e3bef4ab08a8435d9b37b89a7&role.id=43ca25467bc6484da3a50508efb79230 => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:32.232831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952cd614-8aae-4972-ae19-635e5994db4d None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:32.233359 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952cd614-8aae-4972-ae19-635e5994db4d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:32.233359 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952cd614-8aae-4972-ae19-635e5994db4d None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:32.234745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 566/1162] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:32.240077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1ffeb7-34ed-4737-ae4f-700b0adc5d60 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:32.240504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1ffeb7-34ed-4737-ae4f-700b0adc5d60 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:32.240632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1ffeb7-34ed-4737-ae4f-700b0adc5d60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:32.258689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-2c1ffeb7-34ed-4737-ae4f-700b0adc5d60 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:32.259879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-2c1ffeb7-34ed-4737-ae4f-700b0adc5d60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:32.294645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:32.324142 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 597/1163] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:32.591332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952cd614-8aae-4972-ae19-635e5994db4d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:32.691256 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952cd614-8aae-4972-ae19-635e5994db4d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:32.723765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225975a5-7fd9-476e-bf11-ccce79c6518b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:32.724349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225975a5-7fd9-476e-bf11-ccce79c6518b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:32.724636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225975a5-7fd9-476e-bf11-ccce79c6518b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:32.724889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225975a5-7fd9-476e-bf11-ccce79c6518b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:32.742161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:32.790326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225975a5-7fd9-476e-bf11-ccce79c6518b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:32.791748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225975a5-7fd9-476e-bf11-ccce79c6518b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:32.886274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 567/1164] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:28:32 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:28:32.892889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1ffeb7-34ed-4737-ae4f-700b0adc5d60 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:32.989558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1ffeb7-34ed-4737-ae4f-700b0adc5d60 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:33.022967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c4189b-0355-4301-ab28-a8af9bce6bc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:33.023583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c4189b-0355-4301-ab28-a8af9bce6bc1 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:33.023864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c4189b-0355-4301-ab28-a8af9bce6bc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:33.024137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c4189b-0355-4301-ab28-a8af9bce6bc1 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:33.030588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c4189b-0355-4301-ab28-a8af9bce6bc1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:33.032016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c4189b-0355-4301-ab28-a8af9bce6bc1 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-67c4189b-0355-4301-ab28-a8af9bce6bc1 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:28:33.043096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:28:33.044311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:28:33.045367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 598/1165] 10.222.0.65 () {56 vars in 1117 bytes} [Wed Aug 24 03:28:32 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:28:33.051041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225975a5-7fd9-476e-bf11-ccce79c6518b None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:33.097868 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:33.151747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225975a5-7fd9-476e-bf11-ccce79c6518b None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:33.194016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3e1257-2405-48eb-9090-1d32cf9d70e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:33.195043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3e1257-2405-48eb-9090-1d32cf9d70e3 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:33.195535 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3e1257-2405-48eb-9090-1d32cf9d70e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:33.195951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3e1257-2405-48eb-9090-1d32cf9d70e3 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:33.196799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3e1257-2405-48eb-9090-1d32cf9d70e3 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:33.197110 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3e1257-2405-48eb-9090-1d32cf9d70e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:33.198993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3e1257-2405-48eb-9090-1d32cf9d70e3 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:33.207768 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 568/1166] 10.222.0.65 () {56 vars in 1115 bytes} [Wed Aug 24 03:28:33 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:35.099538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6196c2a3-734a-4924-ab6b-bbd147acaefc None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:35.100094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6196c2a3-734a-4924-ab6b-bbd147acaefc None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:35.100094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6196c2a3-734a-4924-ab6b-bbd147acaefc None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:35.101403 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 599/1167] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:35.106613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739db946-c662-42cc-852d-8f6558902165 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:35.106989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739db946-c662-42cc-852d-8f6558902165 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:35.107164 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739db946-c662-42cc-852d-8f6558902165 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:35.126158 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-739db946-c662-42cc-852d-8f6558902165 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:35.127713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-739db946-c662-42cc-852d-8f6558902165 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:35.193992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 569/1168] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:35.464328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6196c2a3-734a-4924-ab6b-bbd147acaefc None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:35.495559 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:35.564010 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6196c2a3-734a-4924-ab6b-bbd147acaefc None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:35.596849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0162a0-a56b-4ed4-923d-96bd399b2862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:35.597443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0162a0-a56b-4ed4-923d-96bd399b2862 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:35.597722 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0162a0-a56b-4ed4-923d-96bd399b2862 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:35.597993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0162a0-a56b-4ed4-923d-96bd399b2862 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:35.598685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0162a0-a56b-4ed4-923d-96bd399b2862 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:35.601009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0162a0-a56b-4ed4-923d-96bd399b2862 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:35.677133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 600/1169] 10.222.0.65 () {60 vars in 1138 bytes} [Wed Aug 24 03:28:35 2022] POST /identity/v3/domains => generated 267 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:28:37.593805 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c0d89f-dcb2-4a24-b44e-cf2da768b7f7 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:37.594408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c0d89f-dcb2-4a24-b44e-cf2da768b7f7 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:37.594506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c0d89f-dcb2-4a24-b44e-cf2da768b7f7 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:37.596385 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 570/1170] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:37.601596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99ec19d-9d0e-46ed-8f40-155290ec8f96 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:37.601997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99ec19d-9d0e-46ed-8f40-155290ec8f96 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:37.602166 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99ec19d-9d0e-46ed-8f40-155290ec8f96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:37.621080 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-f99ec19d-9d0e-46ed-8f40-155290ec8f96 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:37.622298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f99ec19d-9d0e-46ed-8f40-155290ec8f96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:37.688260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 601/1171] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:37.959933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c0d89f-dcb2-4a24-b44e-cf2da768b7f7 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:38.002841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:38.057218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c0d89f-dcb2-4a24-b44e-cf2da768b7f7 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:38.090173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a4503c-8cac-4fb3-997b-62fd6ece85c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:38.090809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a4503c-8cac-4fb3-997b-62fd6ece85c0 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:38.091085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a4503c-8cac-4fb3-997b-62fd6ece85c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:38.091353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a4503c-8cac-4fb3-997b-62fd6ece85c0 None admin] PATH_INFO: `/v3/domains/9431f79495dc4ffabc656a9bfa0a56aa` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:38.101527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a4503c-8cac-4fb3-997b-62fd6ece85c0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=9431f79495dc4ffabc656a9bfa0a56aa)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:38.103072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a4503c-8cac-4fb3-997b-62fd6ece85c0 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:38.104608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 571/1172] 10.222.0.65 () {56 vars in 1183 bytes} [Wed Aug 24 03:28:37 2022] GET /identity/v3/domains/9431f79495dc4ffabc656a9bfa0a56aa => generated 267 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:38.110438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99ec19d-9d0e-46ed-8f40-155290ec8f96 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:38.133905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:38.211128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99ec19d-9d0e-46ed-8f40-155290ec8f96 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:38.243828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8b8646-5136-413f-b196-a65b155426c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:38.244422 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8b8646-5136-413f-b196-a65b155426c5 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:38.244701 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8b8646-5136-413f-b196-a65b155426c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:38.244973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8b8646-5136-413f-b196-a65b155426c5 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:38.245618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8b8646-5136-413f-b196-a65b155426c5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:38.247065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8b8646-5136-413f-b196-a65b155426c5 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:38.420177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 602/1173] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:28:38 2022] POST /identity/v3/projects => generated 361 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:28:40.335508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea815a27-e55f-49f3-8580-6caf880c79aa None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:40.336055 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea815a27-e55f-49f3-8580-6caf880c79aa None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:40.336055 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea815a27-e55f-49f3-8580-6caf880c79aa None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:40.337407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 572/1174] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:40.342708 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe62b1e-4e9e-4631-a41e-5f7728e92a15 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:40.343129 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe62b1e-4e9e-4631-a41e-5f7728e92a15 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:40.343259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe62b1e-4e9e-4631-a41e-5f7728e92a15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:40.361193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-dbe62b1e-4e9e-4631-a41e-5f7728e92a15 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:40.362497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-dbe62b1e-4e9e-4631-a41e-5f7728e92a15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:40.428797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 603/1175] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:40.696247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea815a27-e55f-49f3-8580-6caf880c79aa None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:40.795238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea815a27-e55f-49f3-8580-6caf880c79aa None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:40.796708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:40.828294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4b1d1c-37bd-4817-95e1-bae981619825 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:40.828906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4b1d1c-37bd-4817-95e1-bae981619825 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:40.829185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4b1d1c-37bd-4817-95e1-bae981619825 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:40.829492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4b1d1c-37bd-4817-95e1-bae981619825 None admin] PATH_INFO: `/v3/domains/9431f79495dc4ffabc656a9bfa0a56aa` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:40.830684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4b1d1c-37bd-4817-95e1-bae981619825 None admin] RBAC: Authorizing `identity:get_domain(domain_id=9431f79495dc4ffabc656a9bfa0a56aa)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:40.832100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4b1d1c-37bd-4817-95e1-bae981619825 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:40.833645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 573/1176] 10.222.0.65 () {56 vars in 1183 bytes} [Wed Aug 24 03:28:40 2022] GET /identity/v3/domains/9431f79495dc4ffabc656a9bfa0a56aa => generated 267 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:40.839418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe62b1e-4e9e-4631-a41e-5f7728e92a15 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:40.842456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:40.940220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe62b1e-4e9e-4631-a41e-5f7728e92a15 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:40.973280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecf33ce-da2f-42ea-b1dc-6148574414fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:40.973811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecf33ce-da2f-42ea-b1dc-6148574414fe None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:40.974089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecf33ce-da2f-42ea-b1dc-6148574414fe None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:40.974381 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecf33ce-da2f-42ea-b1dc-6148574414fe None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:40.975011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ecf33ce-da2f-42ea-b1dc-6148574414fe None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:40.976483 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ecf33ce-da2f-42ea-b1dc-6148574414fe None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:41.041035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 604/1177] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:28:40 2022] POST /identity/v3/users => generated 311 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:28:42.982469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d0a502-9846-459b-bb57-4b95350e023b None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:42.983013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d0a502-9846-459b-bb57-4b95350e023b None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:42.983013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d0a502-9846-459b-bb57-4b95350e023b None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:42.984349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 574/1178] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:42.989812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece97165-f4ac-4bae-b0ae-1807a70b93db None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:42.990259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece97165-f4ac-4bae-b0ae-1807a70b93db None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:42.990393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece97165-f4ac-4bae-b0ae-1807a70b93db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:43.008576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-ece97165-f4ac-4bae-b0ae-1807a70b93db None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:43.009769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ece97165-f4ac-4bae-b0ae-1807a70b93db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:43.063979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:43.075290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 605/1179] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:43.344841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d0a502-9846-459b-bb57-4b95350e023b None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:43.445799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d0a502-9846-459b-bb57-4b95350e023b None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:43.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0376de3f-7e72-4e4d-af57-a67a59a9be92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:43.479190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0376de3f-7e72-4e4d-af57-a67a59a9be92 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:43.479466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0376de3f-7e72-4e4d-af57-a67a59a9be92 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:43.479734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0376de3f-7e72-4e4d-af57-a67a59a9be92 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:43.489938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0376de3f-7e72-4e4d-af57-a67a59a9be92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-0376de3f-7e72-4e4d-af57-a67a59a9be92 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:28:43.494651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:28:43.495828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:28:43.496944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 575/1180] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:28:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:28:43.502358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece97165-f4ac-4bae-b0ae-1807a70b93db None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:43.596118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece97165-f4ac-4bae-b0ae-1807a70b93db None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:43.629265 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977edef9-9a05-431b-978d-96b856ead27c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:43.629863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977edef9-9a05-431b-978d-96b856ead27c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:43.630141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977edef9-9a05-431b-978d-96b856ead27c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:43.630452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977edef9-9a05-431b-978d-96b856ead27c None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:43.631374 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977edef9-9a05-431b-978d-96b856ead27c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:43.631677 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977edef9-9a05-431b-978d-96b856ead27c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:43.633042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977edef9-9a05-431b-978d-96b856ead27c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:43.640775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 606/1181] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:28:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:43.646461 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0376de3f-7e72-4e4d-af57-a67a59a9be92 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:43.677949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:43.745116 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0376de3f-7e72-4e4d-af57-a67a59a9be92 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:43.778688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8020349f-0ca8-4440-b11a-97291429590a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:43.779302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8020349f-0ca8-4440-b11a-97291429590a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:43.779579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8020349f-0ca8-4440-b11a-97291429590a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:43.779848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8020349f-0ca8-4440-b11a-97291429590a None admin] PATH_INFO: `/v3/users/1ebca317653043ffb010ca4cbfdbb508` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:43.792840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8020349f-0ca8-4440-b11a-97291429590a None admin] RBAC: Authorizing `identity:get_user(user_id=1ebca317653043ffb010ca4cbfdbb508)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:43.794295 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8020349f-0ca8-4440-b11a-97291429590a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:43.795859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 576/1182] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:43 2022] GET /identity/v3/users/1ebca317653043ffb010ca4cbfdbb508 => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:43.802205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977edef9-9a05-431b-978d-96b856ead27c None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:43.853570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:43.900116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977edef9-9a05-431b-978d-96b856ead27c None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:43.932164 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073931ef-d3cc-498f-8fa1-8b81f7e7a48c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:43.932763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073931ef-d3cc-498f-8fa1-8b81f7e7a48c None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:43.933042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073931ef-d3cc-498f-8fa1-8b81f7e7a48c None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:43.933313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073931ef-d3cc-498f-8fa1-8b81f7e7a48c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:43.997903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073931ef-d3cc-498f-8fa1-8b81f7e7a48c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:43.999378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073931ef-d3cc-498f-8fa1-8b81f7e7a48c None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:44.049427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:44.095090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 607/1183] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:28:43 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:28:44.101764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8020349f-0ca8-4440-b11a-97291429590a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:44.186579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:44.197934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8020349f-0ca8-4440-b11a-97291429590a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:44.231344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e82134-a485-4610-b859-03d948631a9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:44.231951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e82134-a485-4610-b859-03d948631a9b None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:44.232226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e82134-a485-4610-b859-03d948631a9b None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:44.232494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e82134-a485-4610-b859-03d948631a9b None admin] PATH_INFO: `/v3/projects/04529d7d362e4aad96c372842afa3e37` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:44.233591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e82134-a485-4610-b859-03d948631a9b None admin] RBAC: Authorizing `identity:get_project(project_id=04529d7d362e4aad96c372842afa3e37)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:44.235030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e82134-a485-4610-b859-03d948631a9b None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:44.236848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 577/1184] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:28:44 2022] GET /identity/v3/projects/04529d7d362e4aad96c372842afa3e37 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:44.242963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073931ef-d3cc-498f-8fa1-8b81f7e7a48c None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:44.338120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073931ef-d3cc-498f-8fa1-8b81f7e7a48c None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:44.370936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb10a76c-b603-4ba5-bcd4-46df2596eca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:44.371520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb10a76c-b603-4ba5-bcd4-46df2596eca2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:44.371800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb10a76c-b603-4ba5-bcd4-46df2596eca2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:44.372071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb10a76c-b603-4ba5-bcd4-46df2596eca2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:44.372784 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb10a76c-b603-4ba5-bcd4-46df2596eca2 None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=04529d7d362e4aad96c372842afa3e37, user.id=1ebca317653043ffb010ca4cbfdbb508) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:44.373087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb10a76c-b603-4ba5-bcd4-46df2596eca2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:44.374524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb10a76c-b603-4ba5-bcd4-46df2596eca2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:44.379862 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 608/1185] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:28:44 2022] GET /identity/v3/role_assignments?user.id=1ebca317653043ffb010ca4cbfdbb508&scope.project.id=04529d7d362e4aad96c372842afa3e37&role.id=43ca25467bc6484da3a50508efb79230 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:46.260127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef21b97-8ab9-4439-893b-a3d0fac49325 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:46.260678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef21b97-8ab9-4439-893b-a3d0fac49325 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:46.260678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef21b97-8ab9-4439-893b-a3d0fac49325 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:46.262003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 578/1186] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:46.267100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd4d775-76a8-4016-ae74-7dd49b8c66b2 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:46.267618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd4d775-76a8-4016-ae74-7dd49b8c66b2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:46.267889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd4d775-76a8-4016-ae74-7dd49b8c66b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:46.285934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-9fd4d775-76a8-4016-ae74-7dd49b8c66b2 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:46.287171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9fd4d775-76a8-4016-ae74-7dd49b8c66b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:46.297520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:46.352493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 609/1187] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:46.620552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef21b97-8ab9-4439-893b-a3d0fac49325 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:46.719240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef21b97-8ab9-4439-893b-a3d0fac49325 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:46.752655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2190e0ec-f1d8-4f38-b7fe-b9b3d675dc7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:46.753265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2190e0ec-f1d8-4f38-b7fe-b9b3d675dc7c None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:46.753541 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2190e0ec-f1d8-4f38-b7fe-b9b3d675dc7c None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:46.753806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2190e0ec-f1d8-4f38-b7fe-b9b3d675dc7c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:46.763822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2190e0ec-f1d8-4f38-b7fe-b9b3d675dc7c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-2190e0ec-f1d8-4f38-b7fe-b9b3d675dc7c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:28:46.768554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:28:46.769678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:28:46.770904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 579/1188] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:28:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:28:46.775553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd4d775-76a8-4016-ae74-7dd49b8c66b2 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:46.871837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd4d775-76a8-4016-ae74-7dd49b8c66b2 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:46.904447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abc08d3-f019-4d6d-854e-f8094f48f474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:46.905037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abc08d3-f019-4d6d-854e-f8094f48f474 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:46.905319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abc08d3-f019-4d6d-854e-f8094f48f474 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:46.905592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abc08d3-f019-4d6d-854e-f8094f48f474 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:46.906305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abc08d3-f019-4d6d-854e-f8094f48f474 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:46.906606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abc08d3-f019-4d6d-854e-f8094f48f474 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:46.908511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abc08d3-f019-4d6d-854e-f8094f48f474 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:46.916238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 610/1189] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:28:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:46.921878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2190e0ec-f1d8-4f38-b7fe-b9b3d675dc7c None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:46.940756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:47.021878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2190e0ec-f1d8-4f38-b7fe-b9b3d675dc7c None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:47.054481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedddf9b-c3eb-40a6-a5f7-e9e78624c894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:47.055100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedddf9b-c3eb-40a6-a5f7-e9e78624c894 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:47.055378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedddf9b-c3eb-40a6-a5f7-e9e78624c894 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:47.055646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedddf9b-c3eb-40a6-a5f7-e9e78624c894 None admin] PATH_INFO: `/v3/users/1ebca317653043ffb010ca4cbfdbb508` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:47.056747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedddf9b-c3eb-40a6-a5f7-e9e78624c894 None admin] RBAC: Authorizing `identity:get_user(user_id=1ebca317653043ffb010ca4cbfdbb508)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:47.058212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedddf9b-c3eb-40a6-a5f7-e9e78624c894 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:47.059772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 580/1190] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:46 2022] GET /identity/v3/users/1ebca317653043ffb010ca4cbfdbb508 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:47.066133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abc08d3-f019-4d6d-854e-f8094f48f474 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:47.077176 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:47.164012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abc08d3-f019-4d6d-854e-f8094f48f474 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:47.196347 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d46523-9fdf-40da-884b-48490dfc20f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:47.196960 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d46523-9fdf-40da-884b-48490dfc20f4 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:47.197239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d46523-9fdf-40da-884b-48490dfc20f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:47.197513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d46523-9fdf-40da-884b-48490dfc20f4 None admin] PATH_INFO: `/v3/projects/04529d7d362e4aad96c372842afa3e37` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:47.198671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d46523-9fdf-40da-884b-48490dfc20f4 None admin] RBAC: Authorizing `identity:get_project(project_id=04529d7d362e4aad96c372842afa3e37)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:47.200092 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d46523-9fdf-40da-884b-48490dfc20f4 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:47.201683 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 611/1191] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:28:47 2022] GET /identity/v3/projects/04529d7d362e4aad96c372842afa3e37 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:47.208150 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedddf9b-c3eb-40a6-a5f7-e9e78624c894 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:47.290355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:47.305524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedddf9b-c3eb-40a6-a5f7-e9e78624c894 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:47.338812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed194b6c-6b44-4700-8b0a-c53ea2c46e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:47.339450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed194b6c-6b44-4700-8b0a-c53ea2c46e06 None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:47.339727 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed194b6c-6b44-4700-8b0a-c53ea2c46e06 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:47.339999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed194b6c-6b44-4700-8b0a-c53ea2c46e06 None admin] PATH_INFO: `/v3/projects/04529d7d362e4aad96c372842afa3e37/users/1ebca317653043ffb010ca4cbfdbb508/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:47.341892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed194b6c-6b44-4700-8b0a-c53ea2c46e06 None admin] RBAC: Authorizing `identity:create_grant(project_id=04529d7d362e4aad96c372842afa3e37, user_id=1ebca317653043ffb010ca4cbfdbb508, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:47.343340 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed194b6c-6b44-4700-8b0a-c53ea2c46e06 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:47.406860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 581/1192] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:28:47 2022] PUT /identity/v3/projects/04529d7d362e4aad96c372842afa3e37/users/1ebca317653043ffb010ca4cbfdbb508/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:28:49.283485 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fdd22d-dd61-48cf-8575-c479bcdb49f0 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:49.284039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fdd22d-dd61-48cf-8575-c479bcdb49f0 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:49.284039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fdd22d-dd61-48cf-8575-c479bcdb49f0 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:49.285369 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 612/1193] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:28:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:28:49.290843 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e973675-1dfb-4317-bc30-ef711bb2beb9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:49.291284 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e973675-1dfb-4317-bc30-ef711bb2beb9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:49.291382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e973675-1dfb-4317-bc30-ef711bb2beb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:49.309652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1e973675-1dfb-4317-bc30-ef711bb2beb9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:28:49.310909 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1e973675-1dfb-4317-bc30-ef711bb2beb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:28:49.375332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 582/1194] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:28:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:28:49.642890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fdd22d-dd61-48cf-8575-c479bcdb49f0 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:49.720986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:49.743847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fdd22d-dd61-48cf-8575-c479bcdb49f0 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:49.776705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ea1561-fccf-4cde-88bd-4122331f7411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:49.777313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ea1561-fccf-4cde-88bd-4122331f7411 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:49.777585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ea1561-fccf-4cde-88bd-4122331f7411 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:49.777848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ea1561-fccf-4cde-88bd-4122331f7411 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:49.788955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ea1561-fccf-4cde-88bd-4122331f7411 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-d7ea1561-fccf-4cde-88bd-4122331f7411 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:28:49.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:28:49.794715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:28:49.795973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 613/1195] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:28:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:28:49.801531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e973675-1dfb-4317-bc30-ef711bb2beb9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:49.873118 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:49.897696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e973675-1dfb-4317-bc30-ef711bb2beb9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:49.932895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc3e119-e8d7-4e80-a30e-d0ed6d3e60ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:49.933484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc3e119-e8d7-4e80-a30e-d0ed6d3e60ba None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:49.933760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc3e119-e8d7-4e80-a30e-d0ed6d3e60ba None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:49.934031 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc3e119-e8d7-4e80-a30e-d0ed6d3e60ba None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:49.934720 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc3e119-e8d7-4e80-a30e-d0ed6d3e60ba None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:49.935022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc3e119-e8d7-4e80-a30e-d0ed6d3e60ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:49.936378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc3e119-e8d7-4e80-a30e-d0ed6d3e60ba None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:49.944339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 583/1196] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:28:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:28:49.950061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ea1561-fccf-4cde-88bd-4122331f7411 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:50.047395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ea1561-fccf-4cde-88bd-4122331f7411 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:50.079874 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941dfdc3-41f1-4fcb-92d7-248f6ccf5bb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:50.080482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941dfdc3-41f1-4fcb-92d7-248f6ccf5bb3 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:50.080762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941dfdc3-41f1-4fcb-92d7-248f6ccf5bb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:50.081035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941dfdc3-41f1-4fcb-92d7-248f6ccf5bb3 None admin] PATH_INFO: `/v3/users/1ebca317653043ffb010ca4cbfdbb508` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:50.082182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941dfdc3-41f1-4fcb-92d7-248f6ccf5bb3 None admin] RBAC: Authorizing `identity:get_user(user_id=1ebca317653043ffb010ca4cbfdbb508)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:50.083595 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941dfdc3-41f1-4fcb-92d7-248f6ccf5bb3 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:50.085132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 614/1197] 10.222.0.65 () {56 vars in 1177 bytes} [Wed Aug 24 03:28:49 2022] GET /identity/v3/users/1ebca317653043ffb010ca4cbfdbb508 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:50.091023 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc3e119-e8d7-4e80-a30e-d0ed6d3e60ba None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:50.187170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc3e119-e8d7-4e80-a30e-d0ed6d3e60ba None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:50.219918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4a24b5-d8db-41af-9551-69ebbba9b1cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:50.221318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4a24b5-d8db-41af-9551-69ebbba9b1cf None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:50.221803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4a24b5-d8db-41af-9551-69ebbba9b1cf None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:50.222324 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4a24b5-d8db-41af-9551-69ebbba9b1cf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:50.226534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:50.289474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4a24b5-d8db-41af-9551-69ebbba9b1cf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:50.290944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4a24b5-d8db-41af-9551-69ebbba9b1cf None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:50.385671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 584/1198] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:28:50 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:28:50.391649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941dfdc3-41f1-4fcb-92d7-248f6ccf5bb3 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:50.395865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:50.490223 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941dfdc3-41f1-4fcb-92d7-248f6ccf5bb3 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:50.523059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11b7992-f261-40da-a8cf-59b739aa76de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:50.523681 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11b7992-f261-40da-a8cf-59b739aa76de None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:50.523959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11b7992-f261-40da-a8cf-59b739aa76de None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:50.524228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11b7992-f261-40da-a8cf-59b739aa76de None admin] PATH_INFO: `/v3/projects/04529d7d362e4aad96c372842afa3e37` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:50.525336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11b7992-f261-40da-a8cf-59b739aa76de None admin] RBAC: Authorizing `identity:get_project(project_id=04529d7d362e4aad96c372842afa3e37)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:50.526790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11b7992-f261-40da-a8cf-59b739aa76de None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:50.528388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 615/1199] 10.222.0.65 () {56 vars in 1186 bytes} [Wed Aug 24 03:28:50 2022] GET /identity/v3/projects/04529d7d362e4aad96c372842afa3e37 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:28:50.535387 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4a24b5-d8db-41af-9551-69ebbba9b1cf None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:28:50.568829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:28:50.631971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4a24b5-d8db-41af-9551-69ebbba9b1cf None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:28:50.666229 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caacac53-9072-49f2-bf15-2f190b5f1057 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:28:50.666820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caacac53-9072-49f2-bf15-2f190b5f1057 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:28:50.667100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caacac53-9072-49f2-bf15-2f190b5f1057 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:28:50.667371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caacac53-9072-49f2-bf15-2f190b5f1057 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:28:50.668085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caacac53-9072-49f2-bf15-2f190b5f1057 None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=04529d7d362e4aad96c372842afa3e37, user.id=1ebca317653043ffb010ca4cbfdbb508) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:28:50.668392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caacac53-9072-49f2-bf15-2f190b5f1057 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:28:50.669804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caacac53-9072-49f2-bf15-2f190b5f1057 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:28:50.676121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 585/1200] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:28:50 2022] GET /identity/v3/role_assignments?user.id=1ebca317653043ffb010ca4cbfdbb508&scope.project.id=04529d7d362e4aad96c372842afa3e37&role.id=43ca25467bc6484da3a50508efb79230 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:29:00.518979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=64 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:29:00.672946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=50 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:16.791706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8874d6-a955-45ce-b20d-386bf56cc581 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:16.792257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8874d6-a955-45ce-b20d-386bf56cc581 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:16.792257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8874d6-a955-45ce-b20d-386bf56cc581 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:16.793481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 616/1201] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:30:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:30:16.799555 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbdecd6-2cd6-40a1-9095-2a7dee1b287f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:16.800012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbdecd6-2cd6-40a1-9095-2a7dee1b287f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:16.800131 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbdecd6-2cd6-40a1-9095-2a7dee1b287f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:16.818418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-ddbdecd6-2cd6-40a1-9095-2a7dee1b287f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:30:16.819643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ddbdecd6-2cd6-40a1-9095-2a7dee1b287f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:30:16.889047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 586/1202] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:30:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:30:17.161798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8874d6-a955-45ce-b20d-386bf56cc581 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:17.265842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8874d6-a955-45ce-b20d-386bf56cc581 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:17.300558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017b2601-4941-4919-a597-288e4248f9e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:17.301184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017b2601-4941-4919-a597-288e4248f9e4 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:17.301465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017b2601-4941-4919-a597-288e4248f9e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:17.301735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017b2601-4941-4919-a597-288e4248f9e4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:17.312288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017b2601-4941-4919-a597-288e4248f9e4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:17.313877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017b2601-4941-4919-a597-288e4248f9e4 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-017b2601-4941-4919-a597-288e4248f9e4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:17.328155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:17.333207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:17.334568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:17.334568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:17.334568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:17.334568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 617/1203] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:17.336310 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbdecd6-2cd6-40a1-9095-2a7dee1b287f None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:17.446232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbdecd6-2cd6-40a1-9095-2a7dee1b287f None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:17.475666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:17.481937 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d71cee-0af8-4ff2-8256-3dd1ab94b96e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:17.482573 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d71cee-0af8-4ff2-8256-3dd1ab94b96e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:17.482842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d71cee-0af8-4ff2-8256-3dd1ab94b96e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:17.483108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d71cee-0af8-4ff2-8256-3dd1ab94b96e None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:17.483740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d71cee-0af8-4ff2-8256-3dd1ab94b96e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:17.484071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d71cee-0af8-4ff2-8256-3dd1ab94b96e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:17.486327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d71cee-0af8-4ff2-8256-3dd1ab94b96e None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:17.500658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 587/1204] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:17.507560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017b2601-4941-4919-a597-288e4248f9e4 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:17.569779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:17.626091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017b2601-4941-4919-a597-288e4248f9e4 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:17.665277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1235a66-540d-4574-86da-3867a6707204 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:17.665986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1235a66-540d-4574-86da-3867a6707204 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:17.666315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1235a66-540d-4574-86da-3867a6707204 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:17.666666 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1235a66-540d-4574-86da-3867a6707204 None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:17.667443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1235a66-540d-4574-86da-3867a6707204 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:17.669155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1235a66-540d-4574-86da-3867a6707204 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:17.822530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 618/1205] 10.222.0.65 () {60 vars in 1132 bytes} [Wed Aug 24 03:30:17 2022] POST /identity/v3/users => generated 251 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:30:20.018967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6044dae8-d2c0-4b41-8ca9-c3050f2f714f None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:20.019572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6044dae8-d2c0-4b41-8ca9-c3050f2f714f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:20.019572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6044dae8-d2c0-4b41-8ca9-c3050f2f714f None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:20.021100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 588/1206] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:30:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:30:20.027527 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ed48cb-dd36-4994-80c0-96b3c98b44a2 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:20.028129 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ed48cb-dd36-4994-80c0-96b3c98b44a2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:20.028129 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ed48cb-dd36-4994-80c0-96b3c98b44a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:20.049855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-34ed48cb-dd36-4994-80c0-96b3c98b44a2 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:30:20.051467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-34ed48cb-dd36-4994-80c0-96b3c98b44a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:30:20.128835 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 619/1207] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:30:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:30:20.429642 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6044dae8-d2c0-4b41-8ca9-c3050f2f714f None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:20.585383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6044dae8-d2c0-4b41-8ca9-c3050f2f714f None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:20.624698 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e183cb-5fbc-4903-98de-765066931998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:20.625320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e183cb-5fbc-4903-98de-765066931998 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:20.625595 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e183cb-5fbc-4903-98de-765066931998 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:20.625887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e183cb-5fbc-4903-98de-765066931998 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:20.636530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e183cb-5fbc-4903-98de-765066931998 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-33e183cb-5fbc-4903-98de-765066931998 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:30:20.641304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:20.642470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:20.643329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 589/1208] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:30:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:20.648828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ed48cb-dd36-4994-80c0-96b3c98b44a2 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:20.695889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:20.759544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ed48cb-dd36-4994-80c0-96b3c98b44a2 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:20.795229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c706dd3-7f3a-41ee-9d98-f5a8bd5e0d8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:20.795836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c706dd3-7f3a-41ee-9d98-f5a8bd5e0d8d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:20.796112 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c706dd3-7f3a-41ee-9d98-f5a8bd5e0d8d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:20.796380 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c706dd3-7f3a-41ee-9d98-f5a8bd5e0d8d None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:20.797046 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c706dd3-7f3a-41ee-9d98-f5a8bd5e0d8d None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:20.797352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c706dd3-7f3a-41ee-9d98-f5a8bd5e0d8d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:20.799383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c706dd3-7f3a-41ee-9d98-f5a8bd5e0d8d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:20.807662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 620/1209] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:30:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:20.823310 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e183cb-5fbc-4903-98de-765066931998 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:20.881177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:20.934047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e183cb-5fbc-4903-98de-765066931998 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:20.977268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394f5eb9-0a32-40d2-9bff-69e8058c9acd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:20.977268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394f5eb9-0a32-40d2-9bff-69e8058c9acd None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:20.977599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394f5eb9-0a32-40d2-9bff-69e8058c9acd None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:20.977765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394f5eb9-0a32-40d2-9bff-69e8058c9acd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:20.988520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394f5eb9-0a32-40d2-9bff-69e8058c9acd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:20.993904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394f5eb9-0a32-40d2-9bff-69e8058c9acd None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-394f5eb9-0a32-40d2-9bff-69e8058c9acd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:21.005224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:21.006955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:21.008375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:21.008375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:21.008375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:21.008375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 590/1210] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:21.013348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c706dd3-7f3a-41ee-9d98-f5a8bd5e0d8d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:21.122918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:21.126912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c706dd3-7f3a-41ee-9d98-f5a8bd5e0d8d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:21.162963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537c0349-8f82-4316-b58c-fa2ae50df4bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:21.163564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537c0349-8f82-4316-b58c-fa2ae50df4bb None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:21.163844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537c0349-8f82-4316-b58c-fa2ae50df4bb None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:21.164118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537c0349-8f82-4316-b58c-fa2ae50df4bb None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:21.164813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537c0349-8f82-4316-b58c-fa2ae50df4bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:21.165145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537c0349-8f82-4316-b58c-fa2ae50df4bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:21.166859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537c0349-8f82-4316-b58c-fa2ae50df4bb None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:21.180459 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 621/1211] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:21.187136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394f5eb9-0a32-40d2-9bff-69e8058c9acd None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:21.294697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394f5eb9-0a32-40d2-9bff-69e8058c9acd None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:21.296270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:21.330602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3815ada0-c3ce-443d-b814-0712d96e4325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:21.331223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3815ada0-c3ce-443d-b814-0712d96e4325 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:21.331498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3815ada0-c3ce-443d-b814-0712d96e4325 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:21.331767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3815ada0-c3ce-443d-b814-0712d96e4325 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:21.338977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3815ada0-c3ce-443d-b814-0712d96e4325 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:21.340444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3815ada0-c3ce-443d-b814-0712d96e4325 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-3815ada0-c3ce-443d-b814-0712d96e4325 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:30:21.352318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:21.353492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:21.354524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:21.354524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:21.354524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:30:21.354524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:30:21.354524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:30:21.354524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:30:21.354524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:30:21.354524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:30:21.354524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 24 03:30:21.354524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:21.354524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 591/1212] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:21 2022] GET /identity/v3/users/placement => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:21.360064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537c0349-8f82-4316-b58c-fa2ae50df4bb None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:21.470971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537c0349-8f82-4316-b58c-fa2ae50df4bb None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:21.507159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a329297e-df67-45b7-92b7-3b3a76b11ddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:21.507756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a329297e-df67-45b7-92b7-3b3a76b11ddd None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:21.508034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a329297e-df67-45b7-92b7-3b3a76b11ddd None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:21.508304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a329297e-df67-45b7-92b7-3b3a76b11ddd None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:21.509033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a329297e-df67-45b7-92b7-3b3a76b11ddd None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:21.509337 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a329297e-df67-45b7-92b7-3b3a76b11ddd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:21.510878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a329297e-df67-45b7-92b7-3b3a76b11ddd None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:21.528553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 622/1213] 10.222.0.65 () {56 vars in 1143 bytes} [Wed Aug 24 03:30:21 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:30:21.535087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3815ada0-c3ce-443d-b814-0712d96e4325 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:21.645741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3815ada0-c3ce-443d-b814-0712d96e4325 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:21.682262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644d0ee7-1149-4555-be9a-cc24dd076283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:21.682943 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644d0ee7-1149-4555-be9a-cc24dd076283 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:21.683296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644d0ee7-1149-4555-be9a-cc24dd076283 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:21.683643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644d0ee7-1149-4555-be9a-cc24dd076283 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:21.715024 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:21.756939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644d0ee7-1149-4555-be9a-cc24dd076283 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:21.760334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644d0ee7-1149-4555-be9a-cc24dd076283 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:21.909418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 592/1214] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:30:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:30:21.916282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a329297e-df67-45b7-92b7-3b3a76b11ddd None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:21.944942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:22.045053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a329297e-df67-45b7-92b7-3b3a76b11ddd None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:22.087816 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7d0afd-bbff-4c8a-af03-d72175b32354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:22.088721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7d0afd-bbff-4c8a-af03-d72175b32354 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:22.089141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7d0afd-bbff-4c8a-af03-d72175b32354 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:22.089560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7d0afd-bbff-4c8a-af03-d72175b32354 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:22.107430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7d0afd-bbff-4c8a-af03-d72175b32354 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:22.115749 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7d0afd-bbff-4c8a-af03-d72175b32354 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-2f7d0afd-bbff-4c8a-af03-d72175b32354 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:22.127458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:22.128966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:22.129914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:22.129914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:22.129914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:22.129914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 623/1215] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:22.135483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644d0ee7-1149-4555-be9a-cc24dd076283 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:22.210887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:22.250575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644d0ee7-1149-4555-be9a-cc24dd076283 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:22.284788 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75da7aba-3191-49d9-984b-01378eaf3c1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:22.285393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75da7aba-3191-49d9-984b-01378eaf3c1a None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:22.285669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75da7aba-3191-49d9-984b-01378eaf3c1a None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:22.285942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75da7aba-3191-49d9-984b-01378eaf3c1a None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:22.286611 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75da7aba-3191-49d9-984b-01378eaf3c1a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:22.286915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75da7aba-3191-49d9-984b-01378eaf3c1a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:22.288343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75da7aba-3191-49d9-984b-01378eaf3c1a None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:22.301255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 593/1216] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:22.307340 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7d0afd-bbff-4c8a-af03-d72175b32354 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:22.383793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:22.418510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7d0afd-bbff-4c8a-af03-d72175b32354 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:22.456520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a097422-dd54-4162-a8c4-fcb1f0ddd771 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:22.457249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a097422-dd54-4162-a8c4-fcb1f0ddd771 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:22.457645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a097422-dd54-4162-a8c4-fcb1f0ddd771 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:22.457990 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a097422-dd54-4162-a8c4-fcb1f0ddd771 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:22.465407 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a097422-dd54-4162-a8c4-fcb1f0ddd771 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:22.466958 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a097422-dd54-4162-a8c4-fcb1f0ddd771 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-2a097422-dd54-4162-a8c4-fcb1f0ddd771 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:30:22.478579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:22.484126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:22.485601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:30:22.485601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:30:22.485601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:30:22.485601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:30:22.485601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:30:22.485601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:30:22.485601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:30:22.485601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:22.485601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 624/1217] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:30:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:22.489166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75da7aba-3191-49d9-984b-01378eaf3c1a None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:22.595663 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75da7aba-3191-49d9-984b-01378eaf3c1a None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:22.618525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:22.630232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404f8dc7-bb07-43cf-be7c-4799d08199cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:22.630836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404f8dc7-bb07-43cf-be7c-4799d08199cb None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:22.631108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404f8dc7-bb07-43cf-be7c-4799d08199cb None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:22.631373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404f8dc7-bb07-43cf-be7c-4799d08199cb None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:22.632309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404f8dc7-bb07-43cf-be7c-4799d08199cb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:22.632640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404f8dc7-bb07-43cf-be7c-4799d08199cb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:22.634095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404f8dc7-bb07-43cf-be7c-4799d08199cb None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:22.646058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 594/1218] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:30:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:22.652293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a097422-dd54-4162-a8c4-fcb1f0ddd771 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:22.761385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a097422-dd54-4162-a8c4-fcb1f0ddd771 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:22.796468 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:22.803172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca513337-620d-4971-be95-0d150491bd15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:22.803869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca513337-620d-4971-be95-0d150491bd15 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:22.804208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca513337-620d-4971-be95-0d150491bd15 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:22.804537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca513337-620d-4971-be95-0d150491bd15 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:22.805438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca513337-620d-4971-be95-0d150491bd15 None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=4b347c9c89a64168b51c439b4a92d647) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:22.805807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca513337-620d-4971-be95-0d150491bd15 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:22.807632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca513337-620d-4971-be95-0d150491bd15 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:22.814697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 625/1219] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:30:22 2022] GET /identity/v3/role_assignments?user.id=4b347c9c89a64168b51c439b4a92d647&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:24.802250 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80a080e-b25a-49af-926e-64c2551a365d None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:24.802840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80a080e-b25a-49af-926e-64c2551a365d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:24.802840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80a080e-b25a-49af-926e-64c2551a365d None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:24.804052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 595/1220] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:30:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:30:24.810117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bbf0c2-1d48-4bdc-a5f6-95ac83826087 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:24.810750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bbf0c2-1d48-4bdc-a5f6-95ac83826087 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:24.810750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bbf0c2-1d48-4bdc-a5f6-95ac83826087 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:24.833026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-98bbf0c2-1d48-4bdc-a5f6-95ac83826087 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:30:24.838756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-98bbf0c2-1d48-4bdc-a5f6-95ac83826087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:30:24.914671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 626/1221] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:30:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:30:25.188956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80a080e-b25a-49af-926e-64c2551a365d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:25.294727 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80a080e-b25a-49af-926e-64c2551a365d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:25.330078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ac3503-7afd-4c98-a345-20999c9e81b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:25.330744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ac3503-7afd-4c98-a345-20999c9e81b6 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:25.331023 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ac3503-7afd-4c98-a345-20999c9e81b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:25.331296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ac3503-7afd-4c98-a345-20999c9e81b6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:25.341556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ac3503-7afd-4c98-a345-20999c9e81b6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-a8ac3503-7afd-4c98-a345-20999c9e81b6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:30:25.347240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:25.348375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:25.349234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 596/1222] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:30:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:30:25.354591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bbf0c2-1d48-4bdc-a5f6-95ac83826087 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:25.462667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bbf0c2-1d48-4bdc-a5f6-95ac83826087 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:25.489683 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:25.501762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0491f83c-46b5-48db-b18b-18208cf88934 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:25.502472 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0491f83c-46b5-48db-b18b-18208cf88934 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:25.502754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0491f83c-46b5-48db-b18b-18208cf88934 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:25.503029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0491f83c-46b5-48db-b18b-18208cf88934 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:25.503734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0491f83c-46b5-48db-b18b-18208cf88934 None admin] RBAC: Adding query filter params (name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:25.504039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0491f83c-46b5-48db-b18b-18208cf88934 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:25.505572 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0491f83c-46b5-48db-b18b-18208cf88934 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:25.514907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 627/1223] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:30:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:30:25.521890 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ac3503-7afd-4c98-a345-20999c9e81b6 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:25.569324 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:25.630927 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ac3503-7afd-4c98-a345-20999c9e81b6 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:25.666583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a34f5ec-6c53-4ca4-9bd1-0c28d8523427 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:25.667221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a34f5ec-6c53-4ca4-9bd1-0c28d8523427 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:25.667571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a34f5ec-6c53-4ca4-9bd1-0c28d8523427 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:25.667841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a34f5ec-6c53-4ca4-9bd1-0c28d8523427 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:25.679048 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a34f5ec-6c53-4ca4-9bd1-0c28d8523427 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:25.680514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a34f5ec-6c53-4ca4-9bd1-0c28d8523427 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-8a34f5ec-6c53-4ca4-9bd1-0c28d8523427 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:25.695267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:25.697060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:25.698216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:25.698216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:25.698216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:25.698216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 597/1224] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:30:25.703649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0491f83c-46b5-48db-b18b-18208cf88934 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:25.813039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0491f83c-46b5-48db-b18b-18208cf88934 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:25.849312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60639f52-f20b-405c-ac42-b7c2cb5fcd20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:25.849913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60639f52-f20b-405c-ac42-b7c2cb5fcd20 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:25.850214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60639f52-f20b-405c-ac42-b7c2cb5fcd20 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:25.850489 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60639f52-f20b-405c-ac42-b7c2cb5fcd20 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:25.851132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60639f52-f20b-405c-ac42-b7c2cb5fcd20 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:25.851430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60639f52-f20b-405c-ac42-b7c2cb5fcd20 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:25.852786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60639f52-f20b-405c-ac42-b7c2cb5fcd20 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:25.865655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 628/1225] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:25.871514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a34f5ec-6c53-4ca4-9bd1-0c28d8523427 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:25.973458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:25.981161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a34f5ec-6c53-4ca4-9bd1-0c28d8523427 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:26.015926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a853339-bc1f-4b40-9f0c-3d338c9b2b86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:26.017092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a853339-bc1f-4b40-9f0c-3d338c9b2b86 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:26.017360 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a853339-bc1f-4b40-9f0c-3d338c9b2b86 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:26.017627 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a853339-bc1f-4b40-9f0c-3d338c9b2b86 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:26.024887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a853339-bc1f-4b40-9f0c-3d338c9b2b86 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:26.026351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a853339-bc1f-4b40-9f0c-3d338c9b2b86 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-1a853339-bc1f-4b40-9f0c-3d338c9b2b86 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:30:26.037632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:26.038983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:26.040861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:26.040861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:26.040861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:30:26.040861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:30:26.040861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:30:26.040861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:30:26.040861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:30:26.040861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:30:26.040861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 24 03:30:26.040861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:26.042329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 598/1226] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:25 2022] GET /identity/v3/users/placement => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:26.044708 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60639f52-f20b-405c-ac42-b7c2cb5fcd20 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:26.092691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:26.145634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60639f52-f20b-405c-ac42-b7c2cb5fcd20 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:26.179193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea70d592-7d4b-4761-b904-7b38223dcb8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:26.179793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea70d592-7d4b-4761-b904-7b38223dcb8a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:26.180070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea70d592-7d4b-4761-b904-7b38223dcb8a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:26.180339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea70d592-7d4b-4761-b904-7b38223dcb8a None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:26.181060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea70d592-7d4b-4761-b904-7b38223dcb8a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:26.181357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea70d592-7d4b-4761-b904-7b38223dcb8a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:26.182810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea70d592-7d4b-4761-b904-7b38223dcb8a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:26.199646 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 629/1227] 10.222.0.65 () {56 vars in 1143 bytes} [Wed Aug 24 03:30:26 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:26.206995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a853339-bc1f-4b40-9f0c-3d338c9b2b86 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:26.309851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a853339-bc1f-4b40-9f0c-3d338c9b2b86 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:26.346204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9811e8bc-8415-4c99-aae6-602f15a7b20f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:26.346751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9811e8bc-8415-4c99-aae6-602f15a7b20f None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:26.347025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9811e8bc-8415-4c99-aae6-602f15a7b20f None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:26.347299 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9811e8bc-8415-4c99-aae6-602f15a7b20f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:26.357895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9811e8bc-8415-4c99-aae6-602f15a7b20f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:26.359362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9811e8bc-8415-4c99-aae6-602f15a7b20f None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-9811e8bc-8415-4c99-aae6-602f15a7b20f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:26.373206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:26.374696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:26.376151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:26.376151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:26.376151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:26.376151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 599/1228] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:26.381046 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea70d592-7d4b-4761-b904-7b38223dcb8a None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:26.499091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea70d592-7d4b-4761-b904-7b38223dcb8a None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:26.501651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:26.535240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66624d2d-5ded-4e73-8270-3e2574aba30d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:26.535848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66624d2d-5ded-4e73-8270-3e2574aba30d None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:26.536124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66624d2d-5ded-4e73-8270-3e2574aba30d None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:26.536394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66624d2d-5ded-4e73-8270-3e2574aba30d None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:26.537044 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66624d2d-5ded-4e73-8270-3e2574aba30d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:26.537343 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66624d2d-5ded-4e73-8270-3e2574aba30d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:26.538908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66624d2d-5ded-4e73-8270-3e2574aba30d None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:26.551233 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 630/1229] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:26.557094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9811e8bc-8415-4c99-aae6-602f15a7b20f None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:26.567984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:26.663375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9811e8bc-8415-4c99-aae6-602f15a7b20f None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:26.702615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffabbdeb-854c-4429-b441-02133388b9b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:26.702615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffabbdeb-854c-4429-b441-02133388b9b7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:26.702615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffabbdeb-854c-4429-b441-02133388b9b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:26.702615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffabbdeb-854c-4429-b441-02133388b9b7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:26.708434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffabbdeb-854c-4429-b441-02133388b9b7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:26.709859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffabbdeb-854c-4429-b441-02133388b9b7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-ffabbdeb-854c-4429-b441-02133388b9b7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:30:26.729804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:26.731181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:26.732547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:30:26.732547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:30:26.732547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:30:26.732547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:30:26.732547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:30:26.732547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:30:26.732547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:30:26.732547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:26.732547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 600/1230] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:30:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:30:26.737592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66624d2d-5ded-4e73-8270-3e2574aba30d None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:26.845659 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66624d2d-5ded-4e73-8270-3e2574aba30d None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:26.882397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2c1f28-7303-4dda-bd3d-ad1228ec2b19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:26.883012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2c1f28-7303-4dda-bd3d-ad1228ec2b19 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:26.883279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2c1f28-7303-4dda-bd3d-ad1228ec2b19 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:26.883573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2c1f28-7303-4dda-bd3d-ad1228ec2b19 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:26.884406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2c1f28-7303-4dda-bd3d-ad1228ec2b19 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:26.884728 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2c1f28-7303-4dda-bd3d-ad1228ec2b19 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:26.886220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2c1f28-7303-4dda-bd3d-ad1228ec2b19 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:26.899563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 631/1231] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:30:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:26.905944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffabbdeb-854c-4429-b441-02133388b9b7 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:26.977682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:27.015653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffabbdeb-854c-4429-b441-02133388b9b7 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:27.053100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2829ebb8-fac5-4bff-bfce-801b67a1be20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:27.053839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2829ebb8-fac5-4bff-bfce-801b67a1be20 None admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:27.054305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2829ebb8-fac5-4bff-bfce-801b67a1be20 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:27.054582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2829ebb8-fac5-4bff-bfce-801b67a1be20 None admin] PATH_INFO: `/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/4b347c9c89a64168b51c439b4a92d647/roles/94618a16f2e6449ba1cd5fbbd7c19f56` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:27.070627 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2829ebb8-fac5-4bff-bfce-801b67a1be20 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c653d5fa4304f298a295ce635efd5e6, user_id=4b347c9c89a64168b51c439b4a92d647, role_id=94618a16f2e6449ba1cd5fbbd7c19f56)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:27.072290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2829ebb8-fac5-4bff-bfce-801b67a1be20 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:27.203874 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 601/1232] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:30:26 2022] PUT /identity/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/4b347c9c89a64168b51c439b4a92d647/roles/94618a16f2e6449ba1cd5fbbd7c19f56 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:30:29.181795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01433b42-3202-476d-86ae-c25033736fff None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:29.182442 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01433b42-3202-476d-86ae-c25033736fff None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:29.182442 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01433b42-3202-476d-86ae-c25033736fff None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:29.183761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 632/1233] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:30:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:30:29.190346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a33338-456c-4645-ab85-ce8efef216ff None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:29.190866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a33338-456c-4645-ab85-ce8efef216ff None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:29.190866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a33338-456c-4645-ab85-ce8efef216ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:29.209555 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-75a33338-456c-4645-ab85-ce8efef216ff None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:30:29.210809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-75a33338-456c-4645-ab85-ce8efef216ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:30:29.284575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 602/1234] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:30:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:30:29.562460 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01433b42-3202-476d-86ae-c25033736fff None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:29.589890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:29.674686 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01433b42-3202-476d-86ae-c25033736fff None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:29.709730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf03187-ff84-4ff9-9f91-a9cf1285d299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:29.710445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf03187-ff84-4ff9-9f91-a9cf1285d299 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:29.710726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf03187-ff84-4ff9-9f91-a9cf1285d299 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:29.710923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf03187-ff84-4ff9-9f91-a9cf1285d299 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:29.721959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf03187-ff84-4ff9-9f91-a9cf1285d299 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-8bf03187-ff84-4ff9-9f91-a9cf1285d299 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:30:29.726833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:29.728328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:29.729188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 633/1235] 10.222.0.65 () {56 vars in 1102 bytes} [Wed Aug 24 03:30:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:29.734665 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a33338-456c-4645-ab85-ce8efef216ff None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:29.785776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:29.838694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a33338-456c-4645-ab85-ce8efef216ff None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:29.881926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92179744-caee-4069-9280-0a81b625e0c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:29.882564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92179744-caee-4069-9280-0a81b625e0c8 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:29.882839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92179744-caee-4069-9280-0a81b625e0c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:29.883108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92179744-caee-4069-9280-0a81b625e0c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:29.883773 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92179744-caee-4069-9280-0a81b625e0c8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:29.884076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92179744-caee-4069-9280-0a81b625e0c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:29.885466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92179744-caee-4069-9280-0a81b625e0c8 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:29.894309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 603/1236] 10.222.0.65 () {56 vars in 1103 bytes} [Wed Aug 24 03:30:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:29.900273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf03187-ff84-4ff9-9f91-a9cf1285d299 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:29.996676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:30.013731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf03187-ff84-4ff9-9f91-a9cf1285d299 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:30.051974 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac06fd3b-123e-460f-b180-0e12004f3261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:30.052663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac06fd3b-123e-460f-b180-0e12004f3261 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:30.052994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac06fd3b-123e-460f-b180-0e12004f3261 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:30.053321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac06fd3b-123e-460f-b180-0e12004f3261 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:30.066192 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac06fd3b-123e-460f-b180-0e12004f3261 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:30.067889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac06fd3b-123e-460f-b180-0e12004f3261 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-ac06fd3b-123e-460f-b180-0e12004f3261 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:30.089306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:30.090495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:30.091599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:30.091599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:30.091599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:30.091599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 634/1237] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:30.096698 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92179744-caee-4069-9280-0a81b625e0c8 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:30.212032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92179744-caee-4069-9280-0a81b625e0c8 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:30.222877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:30.249897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68f1c56-47b5-4921-b4b6-34094e39d2e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:30.251305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68f1c56-47b5-4921-b4b6-34094e39d2e7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:30.251678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68f1c56-47b5-4921-b4b6-34094e39d2e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:30.252037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68f1c56-47b5-4921-b4b6-34094e39d2e7 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:30.252737 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68f1c56-47b5-4921-b4b6-34094e39d2e7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:30.253077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68f1c56-47b5-4921-b4b6-34094e39d2e7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:30.254525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68f1c56-47b5-4921-b4b6-34094e39d2e7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:30.267646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 604/1238] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:30.274371 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac06fd3b-123e-460f-b180-0e12004f3261 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:30.386832 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac06fd3b-123e-460f-b180-0e12004f3261 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:30.426062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdb519e-1d4d-45b5-beee-a64f8ea2f2b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:30.426730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdb519e-1d4d-45b5-beee-a64f8ea2f2b5 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:30.427004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdb519e-1d4d-45b5-beee-a64f8ea2f2b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:30.427273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdb519e-1d4d-45b5-beee-a64f8ea2f2b5 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:30.429982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:30.435290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdb519e-1d4d-45b5-beee-a64f8ea2f2b5 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:30.436733 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdb519e-1d4d-45b5-beee-a64f8ea2f2b5 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-1fdb519e-1d4d-45b5-beee-a64f8ea2f2b5 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:30:30.449813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:30.451060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:30.452194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:30.452194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:30.452194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:30:30.452194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:30:30.452194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:30:30.452194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:30:30.452194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:30:30.452194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:30:30.452194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 24 03:30:30.452194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:30.452194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 635/1239] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:30 2022] GET /identity/v3/users/placement => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:30.457875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68f1c56-47b5-4921-b4b6-34094e39d2e7 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:30.565860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68f1c56-47b5-4921-b4b6-34094e39d2e7 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:30.602669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f0a5a9-048d-4dd3-beb0-f81796f42049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:30.603267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f0a5a9-048d-4dd3-beb0-f81796f42049 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:30.603616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f0a5a9-048d-4dd3-beb0-f81796f42049 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:30.603874 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f0a5a9-048d-4dd3-beb0-f81796f42049 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:30.604534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f0a5a9-048d-4dd3-beb0-f81796f42049 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:30.604835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f0a5a9-048d-4dd3-beb0-f81796f42049 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:30.606271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f0a5a9-048d-4dd3-beb0-f81796f42049 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:30.623653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 605/1240] 10.222.0.65 () {56 vars in 1143 bytes} [Wed Aug 24 03:30:30 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:30.630655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdb519e-1d4d-45b5-beee-a64f8ea2f2b5 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:30.742306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdb519e-1d4d-45b5-beee-a64f8ea2f2b5 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:30.779416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def819f5-edd6-4030-a88d-59ca12f5e11e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:30.780092 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def819f5-edd6-4030-a88d-59ca12f5e11e None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:30.780471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def819f5-edd6-4030-a88d-59ca12f5e11e None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:30.780794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def819f5-edd6-4030-a88d-59ca12f5e11e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:30.848491 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:30.855553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def819f5-edd6-4030-a88d-59ca12f5e11e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:30.857060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def819f5-edd6-4030-a88d-59ca12f5e11e None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:30.960113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 636/1241] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:30:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:30:30.968108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f0a5a9-048d-4dd3-beb0-f81796f42049 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:30.998210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:31.078609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f0a5a9-048d-4dd3-beb0-f81796f42049 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:31.115462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46734cac-f99c-4f20-94ee-1d3ed03b7799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:31.116145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46734cac-f99c-4f20-94ee-1d3ed03b7799 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:31.116417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46734cac-f99c-4f20-94ee-1d3ed03b7799 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:31.116700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46734cac-f99c-4f20-94ee-1d3ed03b7799 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:31.128364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46734cac-f99c-4f20-94ee-1d3ed03b7799 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:31.129786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46734cac-f99c-4f20-94ee-1d3ed03b7799 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-46734cac-f99c-4f20-94ee-1d3ed03b7799 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:31.146598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:31.148396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:31.150356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:31.150356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:31.150356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:31.150356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 606/1242] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:31.154480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def819f5-edd6-4030-a88d-59ca12f5e11e None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:31.259450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:31.265438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def819f5-edd6-4030-a88d-59ca12f5e11e None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:31.303665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271d606f-2c5d-40eb-b356-86a46d9ed849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:31.304332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271d606f-2c5d-40eb-b356-86a46d9ed849 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:31.304633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271d606f-2c5d-40eb-b356-86a46d9ed849 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:31.304907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271d606f-2c5d-40eb-b356-86a46d9ed849 None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:31.305583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271d606f-2c5d-40eb-b356-86a46d9ed849 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:31.305888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271d606f-2c5d-40eb-b356-86a46d9ed849 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:31.307332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271d606f-2c5d-40eb-b356-86a46d9ed849 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:31.320200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 637/1243] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:31.326335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46734cac-f99c-4f20-94ee-1d3ed03b7799 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:31.414462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:31.440162 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46734cac-f99c-4f20-94ee-1d3ed03b7799 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:31.474138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6136418-3973-4fbc-8960-b96cec573287 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:31.474786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6136418-3973-4fbc-8960-b96cec573287 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:31.475066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6136418-3973-4fbc-8960-b96cec573287 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:31.475337 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6136418-3973-4fbc-8960-b96cec573287 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:31.483227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6136418-3973-4fbc-8960-b96cec573287 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:31.484680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6136418-3973-4fbc-8960-b96cec573287 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-a6136418-3973-4fbc-8960-b96cec573287 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:30:31.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:31.497593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:31.498918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:30:31.498918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:30:31.498918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:30:31.498918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:30:31.498918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:30:31.498918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:30:31.498918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:30:31.498918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:31.498918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 607/1244] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:30:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 24 03:30:31.506369 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271d606f-2c5d-40eb-b356-86a46d9ed849 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:31.623259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271d606f-2c5d-40eb-b356-86a46d9ed849 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:31.668256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e475743-49ca-44e6-b44e-ff72ef8f6d62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:31.669035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e475743-49ca-44e6-b44e-ff72ef8f6d62 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:31.669320 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e475743-49ca-44e6-b44e-ff72ef8f6d62 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:31.669595 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e475743-49ca-44e6-b44e-ff72ef8f6d62 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:31.670373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e475743-49ca-44e6-b44e-ff72ef8f6d62 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:31.670745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e475743-49ca-44e6-b44e-ff72ef8f6d62 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:31.672192 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e475743-49ca-44e6-b44e-ff72ef8f6d62 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:31.685127 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 638/1245] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:30:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:31.692396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6136418-3973-4fbc-8960-b96cec573287 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:31.796602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6136418-3973-4fbc-8960-b96cec573287 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:31.836284 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d7eb5b-c33b-41f6-a558-ad0cedf47c2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:31.836883 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d7eb5b-c33b-41f6-a558-ad0cedf47c2f None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:31.837161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d7eb5b-c33b-41f6-a558-ad0cedf47c2f None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:31.837430 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d7eb5b-c33b-41f6-a558-ad0cedf47c2f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:31.838205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d7eb5b-c33b-41f6-a558-ad0cedf47c2f None admin] RBAC: Adding query filter params (role.id=94618a16f2e6449ba1cd5fbbd7c19f56, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=4b347c9c89a64168b51c439b4a92d647) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:31.838527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d7eb5b-c33b-41f6-a558-ad0cedf47c2f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:31.839981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d7eb5b-c33b-41f6-a558-ad0cedf47c2f None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:31.852264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 608/1246] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:30:31 2022] GET /identity/v3/role_assignments?user.id=4b347c9c89a64168b51c439b4a92d647&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=94618a16f2e6449ba1cd5fbbd7c19f56 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:33.908764 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f28f733-8e62-46d5-818a-e6a2488f55f7 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:33.909576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f28f733-8e62-46d5-818a-e6a2488f55f7 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:33.909576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f28f733-8e62-46d5-818a-e6a2488f55f7 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:33.910760 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 639/1247] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:30:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:30:33.917216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a93b0e6-113b-4ac7-9c99-1e636bb3297f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:33.917216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a93b0e6-113b-4ac7-9c99-1e636bb3297f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:33.917216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a93b0e6-113b-4ac7-9c99-1e636bb3297f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:33.924375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:33.936897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1a93b0e6-113b-4ac7-9c99-1e636bb3297f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:30:33.940239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1a93b0e6-113b-4ac7-9c99-1e636bb3297f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:30:34.022100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 609/1248] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:30:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:30:34.314880 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f28f733-8e62-46d5-818a-e6a2488f55f7 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:34.337758 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:34.430786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f28f733-8e62-46d5-818a-e6a2488f55f7 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:34.467797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcb4347-c3f1-45b3-8085-4ec5a15690e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:34.468441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcb4347-c3f1-45b3-8085-4ec5a15690e3 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:34.468724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcb4347-c3f1-45b3-8085-4ec5a15690e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:34.468996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcb4347-c3f1-45b3-8085-4ec5a15690e3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:34.480222 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcb4347-c3f1-45b3-8085-4ec5a15690e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-0dcb4347-c3f1-45b3-8085-4ec5a15690e3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:30:34.485777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:34.486873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:34.487774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 640/1249] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:30:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:34.493494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a93b0e6-113b-4ac7-9c99-1e636bb3297f None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:34.605730 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a93b0e6-113b-4ac7-9c99-1e636bb3297f None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:34.639985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:34.645226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ee2d97-d366-406c-b854-7d16d5a210e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:34.645900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ee2d97-d366-406c-b854-7d16d5a210e5 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:34.646198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ee2d97-d366-406c-b854-7d16d5a210e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:34.646474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ee2d97-d366-406c-b854-7d16d5a210e5 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:34.647177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ee2d97-d366-406c-b854-7d16d5a210e5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:34.647552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ee2d97-d366-406c-b854-7d16d5a210e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:34.648978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ee2d97-d366-406c-b854-7d16d5a210e5 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:34.657732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 610/1250] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:30:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:34.663801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcb4347-c3f1-45b3-8085-4ec5a15690e3 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:34.745242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:34.772980 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcb4347-c3f1-45b3-8085-4ec5a15690e3 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:34.815965 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ccbde3-f7e5-4a60-b42c-8cc41a80bdca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:34.816624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ccbde3-f7e5-4a60-b42c-8cc41a80bdca None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:34.816905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ccbde3-f7e5-4a60-b42c-8cc41a80bdca None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:34.817176 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ccbde3-f7e5-4a60-b42c-8cc41a80bdca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:34.828536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ccbde3-f7e5-4a60-b42c-8cc41a80bdca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:34.831109 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ccbde3-f7e5-4a60-b42c-8cc41a80bdca None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-26ccbde3-f7e5-4a60-b42c-8cc41a80bdca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:34.846780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:34.848199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:34.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:34.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:34.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:34.849493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 641/1251] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:34.854863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ee2d97-d366-406c-b854-7d16d5a210e5 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:34.965613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ee2d97-d366-406c-b854-7d16d5a210e5 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:35.002471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410f04f0-fdc4-4928-bb9d-0813a14393b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:35.003135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410f04f0-fdc4-4928-bb9d-0813a14393b1 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:35.003408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410f04f0-fdc4-4928-bb9d-0813a14393b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:35.003680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410f04f0-fdc4-4928-bb9d-0813a14393b1 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:35.004315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410f04f0-fdc4-4928-bb9d-0813a14393b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:35.004608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410f04f0-fdc4-4928-bb9d-0813a14393b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:35.006011 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410f04f0-fdc4-4928-bb9d-0813a14393b1 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:35.019309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 611/1252] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:35.025574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ccbde3-f7e5-4a60-b42c-8cc41a80bdca None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:35.130411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ccbde3-f7e5-4a60-b42c-8cc41a80bdca None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:35.158513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:35.167949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b44c8f6-58d9-43d7-bc03-050d289f6b73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:35.168914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b44c8f6-58d9-43d7-bc03-050d289f6b73 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:35.169371 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b44c8f6-58d9-43d7-bc03-050d289f6b73 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:35.169821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b44c8f6-58d9-43d7-bc03-050d289f6b73 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:35.179498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b44c8f6-58d9-43d7-bc03-050d289f6b73 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:35.180951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b44c8f6-58d9-43d7-bc03-050d289f6b73 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-7b44c8f6-58d9-43d7-bc03-050d289f6b73 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:30:35.193227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:35.194511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:35.195506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:35.195506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:35.195506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:30:35.195506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:30:35.195506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:30:35.195506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:30:35.195506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:30:35.195506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:30:35.195506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 24 03:30:35.195506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:35.195506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 642/1253] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:35 2022] GET /identity/v3/users/placement => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:35.201022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410f04f0-fdc4-4928-bb9d-0813a14393b1 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:35.268168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:35.317383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410f04f0-fdc4-4928-bb9d-0813a14393b1 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:35.354724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0558fda-aabc-453d-b45c-49bd8b93307e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:35.355327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0558fda-aabc-453d-b45c-49bd8b93307e None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:35.355599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0558fda-aabc-453d-b45c-49bd8b93307e None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:35.355867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0558fda-aabc-453d-b45c-49bd8b93307e None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:35.356724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0558fda-aabc-453d-b45c-49bd8b93307e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:35.357260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0558fda-aabc-453d-b45c-49bd8b93307e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:35.359015 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0558fda-aabc-453d-b45c-49bd8b93307e None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:35.379386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 612/1254] 10.222.0.65 () {56 vars in 1143 bytes} [Wed Aug 24 03:30:35 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:30:35.386237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b44c8f6-58d9-43d7-bc03-050d289f6b73 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:35.506210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b44c8f6-58d9-43d7-bc03-050d289f6b73 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:35.542456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc27055c-93c1-4a76-bff9-3c7689c4e833 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:35.542981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc27055c-93c1-4a76-bff9-3c7689c4e833 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:35.543261 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc27055c-93c1-4a76-bff9-3c7689c4e833 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:35.543531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc27055c-93c1-4a76-bff9-3c7689c4e833 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:35.601638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:35.614455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc27055c-93c1-4a76-bff9-3c7689c4e833 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:35.615923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc27055c-93c1-4a76-bff9-3c7689c4e833 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:35.717908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 643/1255] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:30:35 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:30:35.724397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0558fda-aabc-453d-b45c-49bd8b93307e None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:35.834296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0558fda-aabc-453d-b45c-49bd8b93307e None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:35.849475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:35.873138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217f74a8-c9fd-4caa-a724-d7607069c148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:35.873885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217f74a8-c9fd-4caa-a724-d7607069c148 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:35.874215 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217f74a8-c9fd-4caa-a724-d7607069c148 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:35.874471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217f74a8-c9fd-4caa-a724-d7607069c148 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:35.893005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217f74a8-c9fd-4caa-a724-d7607069c148 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:35.894520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217f74a8-c9fd-4caa-a724-d7607069c148 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-217f74a8-c9fd-4caa-a724-d7607069c148 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:35.909918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:35.911347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:35.912526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:35.912526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:35.912526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:35.912526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 613/1256] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:35.917133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc27055c-93c1-4a76-bff9-3c7689c4e833 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:36.013754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:36.026625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc27055c-93c1-4a76-bff9-3c7689c4e833 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:36.060792 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e046b60d-3204-4640-bd6c-826c539252fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:36.061399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e046b60d-3204-4640-bd6c-826c539252fc None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:36.061681 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e046b60d-3204-4640-bd6c-826c539252fc None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:36.061956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e046b60d-3204-4640-bd6c-826c539252fc None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:36.062639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e046b60d-3204-4640-bd6c-826c539252fc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:36.062946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e046b60d-3204-4640-bd6c-826c539252fc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:36.064550 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e046b60d-3204-4640-bd6c-826c539252fc None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:36.075969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 644/1257] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:36.082434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217f74a8-c9fd-4caa-a724-d7607069c148 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:36.188531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217f74a8-c9fd-4caa-a724-d7607069c148 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:36.226113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4f3915-0efb-4171-82ac-d3d6245d1acf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:36.227022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4f3915-0efb-4171-82ac-d3d6245d1acf None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:36.227391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4f3915-0efb-4171-82ac-d3d6245d1acf None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:36.227801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4f3915-0efb-4171-82ac-d3d6245d1acf None admin] PATH_INFO: `/v3/projects/service` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:36.236096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4f3915-0efb-4171-82ac-d3d6245d1acf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:36.237634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4f3915-0efb-4171-82ac-d3d6245d1acf None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-bb4f3915-0efb-4171-82ac-d3d6245d1acf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:30:36.249621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:36.251275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:36.252682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:30:36.252682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:30:36.252682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:30:36.252682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:30:36.252682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:30:36.252682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:30:36.252682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:30:36.252682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:36.252682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 614/1258] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:30:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:36.258248 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e046b60d-3204-4640-bd6c-826c539252fc None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:36.370525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e046b60d-3204-4640-bd6c-826c539252fc None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:36.408890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630a236f-a958-4294-a02f-aefb1911dd53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:36.411942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630a236f-a958-4294-a02f-aefb1911dd53 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:36.411942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630a236f-a958-4294-a02f-aefb1911dd53 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:36.411942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630a236f-a958-4294-a02f-aefb1911dd53 None admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:36.411942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630a236f-a958-4294-a02f-aefb1911dd53 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:36.411942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630a236f-a958-4294-a02f-aefb1911dd53 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:36.412852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630a236f-a958-4294-a02f-aefb1911dd53 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:36.427469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 645/1259] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:30:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:36.433784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4f3915-0efb-4171-82ac-d3d6245d1acf None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:36.460812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:36.540039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4f3915-0efb-4171-82ac-d3d6245d1acf None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:36.578833 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efaab49-0e3e-4068-a785-87454c841dfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:36.579571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efaab49-0e3e-4068-a785-87454c841dfc None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:36.579951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efaab49-0e3e-4068-a785-87454c841dfc None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:36.580273 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efaab49-0e3e-4068-a785-87454c841dfc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:36.581140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efaab49-0e3e-4068-a785-87454c841dfc None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=4b347c9c89a64168b51c439b4a92d647) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:36.581472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efaab49-0e3e-4068-a785-87454c841dfc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:36.583243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efaab49-0e3e-4068-a785-87454c841dfc None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:36.589419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 615/1260] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:30:36 2022] GET /identity/v3/role_assignments?user.id=4b347c9c89a64168b51c439b4a92d647&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=43ca25467bc6484da3a50508efb79230 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:38.568872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12443cc1-6b0b-4ea6-b8da-80ef8bbea452 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:38.569433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12443cc1-6b0b-4ea6-b8da-80ef8bbea452 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:38.569531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12443cc1-6b0b-4ea6-b8da-80ef8bbea452 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:38.571369 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 646/1261] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:30:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:30:38.578410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c43222-8603-43cd-b778-cbd344ae1de9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:38.579673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c43222-8603-43cd-b778-cbd344ae1de9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:38.579673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c43222-8603-43cd-b778-cbd344ae1de9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:38.599763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-c8c43222-8603-43cd-b778-cbd344ae1de9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:30:38.601115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c8c43222-8603-43cd-b778-cbd344ae1de9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:30:38.678396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 616/1262] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:30:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:30:38.955511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12443cc1-6b0b-4ea6-b8da-80ef8bbea452 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:38.966246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:39.067707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12443cc1-6b0b-4ea6-b8da-80ef8bbea452 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:39.105790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73e5ce0-9126-4fdf-ad8e-4a88f37d6323 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:39.106465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73e5ce0-9126-4fdf-ad8e-4a88f37d6323 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:39.106748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73e5ce0-9126-4fdf-ad8e-4a88f37d6323 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:39.107025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73e5ce0-9126-4fdf-ad8e-4a88f37d6323 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:39.120467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73e5ce0-9126-4fdf-ad8e-4a88f37d6323 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-d73e5ce0-9126-4fdf-ad8e-4a88f37d6323 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:30:39.126859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:39.155570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:39.157484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 647/1263] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:30:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:39.158560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c43222-8603-43cd-b778-cbd344ae1de9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:39.162987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:39.249546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c43222-8603-43cd-b778-cbd344ae1de9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:39.289167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0862738b-dd3d-4fe2-8337-83e9dbf23304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:39.289966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0862738b-dd3d-4fe2-8337-83e9dbf23304 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:39.290270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0862738b-dd3d-4fe2-8337-83e9dbf23304 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:39.290540 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0862738b-dd3d-4fe2-8337-83e9dbf23304 None admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:39.291191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0862738b-dd3d-4fe2-8337-83e9dbf23304 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:39.291489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0862738b-dd3d-4fe2-8337-83e9dbf23304 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:39.292831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0862738b-dd3d-4fe2-8337-83e9dbf23304 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:39.301066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 617/1264] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:30:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:30:39.306945 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73e5ce0-9126-4fdf-ad8e-4a88f37d6323 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:39.378253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:39.420251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73e5ce0-9126-4fdf-ad8e-4a88f37d6323 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:39.455487 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6088b26-4584-4ba7-a048-a8d67f728daa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:39.456116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6088b26-4584-4ba7-a048-a8d67f728daa None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:39.456409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6088b26-4584-4ba7-a048-a8d67f728daa None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:39.456658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6088b26-4584-4ba7-a048-a8d67f728daa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:39.467364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6088b26-4584-4ba7-a048-a8d67f728daa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:39.468794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6088b26-4584-4ba7-a048-a8d67f728daa None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-d6088b26-4584-4ba7-a048-a8d67f728daa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:39.483428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:39.488997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:39.490615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:39.490615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:39.490615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:39.490615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 648/1265] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:39.491406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0862738b-dd3d-4fe2-8337-83e9dbf23304 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:39.584885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:39.606094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0862738b-dd3d-4fe2-8337-83e9dbf23304 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:39.647753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91176918-e484-4b4d-9db9-a9761727b899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:39.648476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91176918-e484-4b4d-9db9-a9761727b899 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:39.648779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91176918-e484-4b4d-9db9-a9761727b899 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:39.649072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91176918-e484-4b4d-9db9-a9761727b899 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:39.649724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91176918-e484-4b4d-9db9-a9761727b899 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:39.650022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91176918-e484-4b4d-9db9-a9761727b899 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:39.651659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91176918-e484-4b4d-9db9-a9761727b899 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:39.665065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 618/1266] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:39.671131 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6088b26-4584-4ba7-a048-a8d67f728daa None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:39.776515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6088b26-4584-4ba7-a048-a8d67f728daa None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:39.800311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:39.816639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3fb41a-028c-47bc-bcdb-8ddddf0357a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:39.817277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3fb41a-028c-47bc-bcdb-8ddddf0357a3 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:39.817554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3fb41a-028c-47bc-bcdb-8ddddf0357a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:39.817838 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3fb41a-028c-47bc-bcdb-8ddddf0357a3 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:39.825809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3fb41a-028c-47bc-bcdb-8ddddf0357a3 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:39.827505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3fb41a-028c-47bc-bcdb-8ddddf0357a3 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-0d3fb41a-028c-47bc-bcdb-8ddddf0357a3 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:30:39.839372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:39.840651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:39.842023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:39.842023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:39.842023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:30:39.842023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:30:39.842023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:30:39.842023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:30:39.842023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:30:39.842023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:30:39.842023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 24 03:30:39.842023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:39.842023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 649/1267] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:39 2022] GET /identity/v3/users/placement => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:39.846898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91176918-e484-4b4d-9db9-a9761727b899 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:39.960096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91176918-e484-4b4d-9db9-a9761727b899 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:39.995646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ea1373-3cea-4ef6-a4c3-6284dbbd3d31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:39.996244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ea1373-3cea-4ef6-a4c3-6284dbbd3d31 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:39.996520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ea1373-3cea-4ef6-a4c3-6284dbbd3d31 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:39.996788 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ea1373-3cea-4ef6-a4c3-6284dbbd3d31 None admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:39.997515 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ea1373-3cea-4ef6-a4c3-6284dbbd3d31 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:39.997821 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ea1373-3cea-4ef6-a4c3-6284dbbd3d31 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:39.999299 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ea1373-3cea-4ef6-a4c3-6284dbbd3d31 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:40.010076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:40.018456 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 619/1268] 10.222.0.65 () {56 vars in 1143 bytes} [Wed Aug 24 03:30:39 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:40.024900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3fb41a-028c-47bc-bcdb-8ddddf0357a3 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:40.143163 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3fb41a-028c-47bc-bcdb-8ddddf0357a3 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:40.184351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ebe0f4-b161-4c29-a363-d4d110ba9000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:40.184952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ebe0f4-b161-4c29-a363-d4d110ba9000 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:40.185224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ebe0f4-b161-4c29-a363-d4d110ba9000 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:40.185494 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ebe0f4-b161-4c29-a363-d4d110ba9000 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:40.196320 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ebe0f4-b161-4c29-a363-d4d110ba9000 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:40.197803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ebe0f4-b161-4c29-a363-d4d110ba9000 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-54ebe0f4-b161-4c29-a363-d4d110ba9000 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:40.212106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:40.213578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:40.219210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:40.219210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:40.219210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:40.219210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 650/1269] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:40.220214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ea1373-3cea-4ef6-a4c3-6284dbbd3d31 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:40.332993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ea1373-3cea-4ef6-a4c3-6284dbbd3d31 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:40.372588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef03b246-1c68-4a67-8132-57a90fe95b00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:40.373246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef03b246-1c68-4a67-8132-57a90fe95b00 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:40.373613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef03b246-1c68-4a67-8132-57a90fe95b00 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:40.373898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef03b246-1c68-4a67-8132-57a90fe95b00 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:40.374582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef03b246-1c68-4a67-8132-57a90fe95b00 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:40.374883 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef03b246-1c68-4a67-8132-57a90fe95b00 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:40.376413 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef03b246-1c68-4a67-8132-57a90fe95b00 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:40.389977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 620/1270] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:30:40.397376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ebe0f4-b161-4c29-a363-d4d110ba9000 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:40.436946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:40.515892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ebe0f4-b161-4c29-a363-d4d110ba9000 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:40.554181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e55dbb-6fac-406c-86b2-05e28ea161e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:40.554840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e55dbb-6fac-406c-86b2-05e28ea161e7 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:40.555116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e55dbb-6fac-406c-86b2-05e28ea161e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:40.555387 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e55dbb-6fac-406c-86b2-05e28ea161e7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:40.562608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e55dbb-6fac-406c-86b2-05e28ea161e7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:40.564304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e55dbb-6fac-406c-86b2-05e28ea161e7 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-38e55dbb-6fac-406c-86b2-05e28ea161e7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:30:40.575984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:40.577277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:40.579254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:30:40.579254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:30:40.579254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:30:40.579254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:30:40.579254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:30:40.579254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:30:40.579254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:30:40.579254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:40.579254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 651/1271] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:30:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:40.583834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef03b246-1c68-4a67-8132-57a90fe95b00 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:40.649556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:40.696301 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef03b246-1c68-4a67-8132-57a90fe95b00 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:40.733578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64d5ab4-7eef-4a16-84f0-d9417af3ca27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:40.734209 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64d5ab4-7eef-4a16-84f0-d9417af3ca27 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:40.734491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64d5ab4-7eef-4a16-84f0-d9417af3ca27 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:40.734762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64d5ab4-7eef-4a16-84f0-d9417af3ca27 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:40.735515 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64d5ab4-7eef-4a16-84f0-d9417af3ca27 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:40.735815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64d5ab4-7eef-4a16-84f0-d9417af3ca27 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:40.737245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64d5ab4-7eef-4a16-84f0-d9417af3ca27 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:40.749953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 621/1272] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:30:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:40.757947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e55dbb-6fac-406c-86b2-05e28ea161e7 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:40.869868 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:40.879358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e55dbb-6fac-406c-86b2-05e28ea161e7 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:40.919440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ad272d-b571-4a26-84ee-e3008cc9e8d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:40.920119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ad272d-b571-4a26-84ee-e3008cc9e8d2 None admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:40.920412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ad272d-b571-4a26-84ee-e3008cc9e8d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:40.920720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ad272d-b571-4a26-84ee-e3008cc9e8d2 None admin] PATH_INFO: `/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/4b347c9c89a64168b51c439b4a92d647/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:40.922971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ad272d-b571-4a26-84ee-e3008cc9e8d2 None admin] RBAC: Authorizing `identity:create_grant(project_id=5c653d5fa4304f298a295ce635efd5e6, user_id=4b347c9c89a64168b51c439b4a92d647, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:40.924644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ad272d-b571-4a26-84ee-e3008cc9e8d2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:41.018833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 652/1273] 10.222.0.65 () {58 vars in 1439 bytes} [Wed Aug 24 03:30:40 2022] PUT /identity/v3/projects/5c653d5fa4304f298a295ce635efd5e6/users/4b347c9c89a64168b51c439b4a92d647/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:30:42.954547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fbd699-40b4-48ce-9868-cf19112005e3 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:42.955142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fbd699-40b4-48ce-9868-cf19112005e3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:42.955142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fbd699-40b4-48ce-9868-cf19112005e3 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:42.956532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 622/1274] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:30:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:30:42.962900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c728905-a04a-47f3-98d4-1179df21cd7c None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:42.963408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c728905-a04a-47f3-98d4-1179df21cd7c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:42.963523 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c728905-a04a-47f3-98d4-1179df21cd7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:42.986289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5c728905-a04a-47f3-98d4-1179df21cd7c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:30:42.987510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5c728905-a04a-47f3-98d4-1179df21cd7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:30:43.059511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 653/1275] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:30:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:30:43.361849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fbd699-40b4-48ce-9868-cf19112005e3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:43.476379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fbd699-40b4-48ce-9868-cf19112005e3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:43.494006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:43.511870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fdaa2c-5ed5-41f6-9f5b-ae088da1fa52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:43.512495 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fdaa2c-5ed5-41f6-9f5b-ae088da1fa52 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:43.512770 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fdaa2c-5ed5-41f6-9f5b-ae088da1fa52 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:43.513037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fdaa2c-5ed5-41f6-9f5b-ae088da1fa52 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:43.523984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fdaa2c-5ed5-41f6-9f5b-ae088da1fa52 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-37fdaa2c-5ed5-41f6-9f5b-ae088da1fa52 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 24 03:30:43.528700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:43.529888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:43.530971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 623/1276] 10.222.0.65 () {56 vars in 1096 bytes} [Wed Aug 24 03:30:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:43.537152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c728905-a04a-47f3-98d4-1179df21cd7c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:43.623060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:43.643879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c728905-a04a-47f3-98d4-1179df21cd7c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:43.682292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c78d8ce-9fff-4812-81d2-96b750d03457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:43.682965 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c78d8ce-9fff-4812-81d2-96b750d03457 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:43.683283 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c78d8ce-9fff-4812-81d2-96b750d03457 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:43.683574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c78d8ce-9fff-4812-81d2-96b750d03457 None admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:43.684359 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c78d8ce-9fff-4812-81d2-96b750d03457 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:43.684664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c78d8ce-9fff-4812-81d2-96b750d03457 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:43.686196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c78d8ce-9fff-4812-81d2-96b750d03457 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:43.695972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 654/1277] 10.222.0.65 () {56 vars in 1099 bytes} [Wed Aug 24 03:30:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:43.702608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fdaa2c-5ed5-41f6-9f5b-ae088da1fa52 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:43.809797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fdaa2c-5ed5-41f6-9f5b-ae088da1fa52 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:43.846237 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad62ad4-1aff-4512-9543-f76451230178 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:43.846868 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad62ad4-1aff-4512-9543-f76451230178 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:43.847146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad62ad4-1aff-4512-9543-f76451230178 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:43.847417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad62ad4-1aff-4512-9543-f76451230178 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:43.858285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad62ad4-1aff-4512-9543-f76451230178 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:43.859854 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad62ad4-1aff-4512-9543-f76451230178 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-dad62ad4-1aff-4512-9543-f76451230178 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:43.874307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:43.875399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:43.877381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:43.877381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:43.877381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:43.877381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 624/1278] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:43.882697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c78d8ce-9fff-4812-81d2-96b750d03457 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:43.997215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c78d8ce-9fff-4812-81d2-96b750d03457 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:44.033234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007bf5c0-ae92-4ba1-8d65-e28a4055d49b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:44.033896 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007bf5c0-ae92-4ba1-8d65-e28a4055d49b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:44.034128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007bf5c0-ae92-4ba1-8d65-e28a4055d49b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:44.034427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007bf5c0-ae92-4ba1-8d65-e28a4055d49b None admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:44.035131 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007bf5c0-ae92-4ba1-8d65-e28a4055d49b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:44.035437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007bf5c0-ae92-4ba1-8d65-e28a4055d49b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:44.036810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007bf5c0-ae92-4ba1-8d65-e28a4055d49b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:44.050977 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 655/1279] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:44.056757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad62ad4-1aff-4512-9543-f76451230178 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:44.091846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:44.172798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad62ad4-1aff-4512-9543-f76451230178 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:44.213221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e16d29a-21d3-448e-b4bf-be13d8af5856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:44.213863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e16d29a-21d3-448e-b4bf-be13d8af5856 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:44.214142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e16d29a-21d3-448e-b4bf-be13d8af5856 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:44.214444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e16d29a-21d3-448e-b4bf-be13d8af5856 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:44.222422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e16d29a-21d3-448e-b4bf-be13d8af5856 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:44.223870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e16d29a-21d3-448e-b4bf-be13d8af5856 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-0e16d29a-21d3-448e-b4bf-be13d8af5856 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 24 03:30:44.236328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:44.237719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:44.239247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:44.239247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:44.239247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 24 03:30:44.239247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 24 03:30:44.239247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 24 03:30:44.239247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 24 03:30:44.239247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 24 03:30:44.239247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 24 03:30:44.239247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 24 03:30:44.239247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:44.239247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 625/1280] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:44 2022] GET /identity/v3/users/placement => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:44.243679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007bf5c0-ae92-4ba1-8d65-e28a4055d49b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:44.247921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:44.352927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007bf5c0-ae92-4ba1-8d65-e28a4055d49b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:44.395091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c558d-f8fd-43d9-ab59-36eb2b7f9e40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:44.395790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c558d-f8fd-43d9-ab59-36eb2b7f9e40 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:44.396120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c558d-f8fd-43d9-ab59-36eb2b7f9e40 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:44.396509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c558d-f8fd-43d9-ab59-36eb2b7f9e40 None admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:44.397301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6c558d-f8fd-43d9-ab59-36eb2b7f9e40 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:44.397606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6c558d-f8fd-43d9-ab59-36eb2b7f9e40 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:44.399241 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6c558d-f8fd-43d9-ab59-36eb2b7f9e40 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:44.419659 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 656/1281] 10.222.0.65 () {56 vars in 1143 bytes} [Wed Aug 24 03:30:44 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:30:44.426059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e16d29a-21d3-448e-b4bf-be13d8af5856 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:44.526472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:44.537352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e16d29a-21d3-448e-b4bf-be13d8af5856 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:44.573519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cd0bda-b7c0-4e14-93be-09af53378c58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:44.574188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cd0bda-b7c0-4e14-93be-09af53378c58 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:44.574472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cd0bda-b7c0-4e14-93be-09af53378c58 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:44.574746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cd0bda-b7c0-4e14-93be-09af53378c58 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:44.645915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cd0bda-b7c0-4e14-93be-09af53378c58 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:44.647527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cd0bda-b7c0-4e14-93be-09af53378c58 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:44.742600 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:44.753163 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 626/1282] 10.222.0.65 () {58 vars in 1282 bytes} [Wed Aug 24 03:30:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 24 03:30:44.759284 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c558d-f8fd-43d9-ab59-36eb2b7f9e40 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:44.825152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:44.865213 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c558d-f8fd-43d9-ab59-36eb2b7f9e40 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:44.900508 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48783df4-f475-4bc4-b44f-e8140b4f65d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:44.901135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48783df4-f475-4bc4-b44f-e8140b4f65d3 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:44.901410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48783df4-f475-4bc4-b44f-e8140b4f65d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:44.901717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48783df4-f475-4bc4-b44f-e8140b4f65d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:44.913373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48783df4-f475-4bc4-b44f-e8140b4f65d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:44.914867 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48783df4-f475-4bc4-b44f-e8140b4f65d3 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-48783df4-f475-4bc4-b44f-e8140b4f65d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 24 03:30:44.933195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:44.937808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 24 03:30:44.939947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 24 03:30:44.939947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 24 03:30:44.939947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:44.939947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 657/1283] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:30:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:44.940822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cd0bda-b7c0-4e14-93be-09af53378c58 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:45.052947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cd0bda-b7c0-4e14-93be-09af53378c58 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:45.088941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622f2fdb-37ed-4ad2-8fc5-f9eae47bd298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:45.089797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622f2fdb-37ed-4ad2-8fc5-f9eae47bd298 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:45.089797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622f2fdb-37ed-4ad2-8fc5-f9eae47bd298 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:45.090063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622f2fdb-37ed-4ad2-8fc5-f9eae47bd298 None admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:45.090733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622f2fdb-37ed-4ad2-8fc5-f9eae47bd298 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:45.091029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622f2fdb-37ed-4ad2-8fc5-f9eae47bd298 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:45.092412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622f2fdb-37ed-4ad2-8fc5-f9eae47bd298 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:45.105100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 627/1284] 10.222.0.65 () {56 vars in 1109 bytes} [Wed Aug 24 03:30:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:45.111125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48783df4-f475-4bc4-b44f-e8140b4f65d3 None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:45.230724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48783df4-f475-4bc4-b44f-e8140b4f65d3 None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:45.246877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:45.268551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b513d88-daa0-4869-9166-b243d3df114b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:45.269188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b513d88-daa0-4869-9166-b243d3df114b None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:45.269466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b513d88-daa0-4869-9166-b243d3df114b None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:45.269767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b513d88-daa0-4869-9166-b243d3df114b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:45.276663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b513d88-daa0-4869-9166-b243d3df114b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:45.278109 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b513d88-daa0-4869-9166-b243d3df114b None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application [None req-2b513d88-daa0-4869-9166-b243d3df114b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 24 03:30:45.290782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:45.294670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:45.295783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 24 03:30:45.295783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 24 03:30:45.295783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 24 03:30:45.295783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 24 03:30:45.295783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 24 03:30:45.295783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 24 03:30:45.295783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 24 03:30:45.295783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: ERROR keystone.server.flask.application Aug 24 03:30:45.295783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 658/1285] 10.222.0.65 () {56 vars in 1111 bytes} [Wed Aug 24 03:30:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:45.298205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622f2fdb-37ed-4ad2-8fc5-f9eae47bd298 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:45.357927 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:45.416500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622f2fdb-37ed-4ad2-8fc5-f9eae47bd298 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:45.454270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8bf22f-2ba4-47e1-94d0-a0f846ba5cf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:45.455241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8bf22f-2ba4-47e1-94d0-a0f846ba5cf8 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:45.455673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8bf22f-2ba4-47e1-94d0-a0f846ba5cf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:45.455966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8bf22f-2ba4-47e1-94d0-a0f846ba5cf8 None admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:45.456723 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8bf22f-2ba4-47e1-94d0-a0f846ba5cf8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:45.457029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8bf22f-2ba4-47e1-94d0-a0f846ba5cf8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:45.458537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8bf22f-2ba4-47e1-94d0-a0f846ba5cf8 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:45.471365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 628/1286] 10.222.0.65 () {56 vars in 1148 bytes} [Wed Aug 24 03:30:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:45.478074 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b513d88-daa0-4869-9166-b243d3df114b None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:45.591447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b513d88-daa0-4869-9166-b243d3df114b None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:45.627454 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0254321b-6a0d-4b22-abf1-fc77443e13f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:45.627967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0254321b-6a0d-4b22-abf1-fc77443e13f2 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:45.628249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0254321b-6a0d-4b22-abf1-fc77443e13f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:45.628522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0254321b-6a0d-4b22-abf1-fc77443e13f2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:45.629287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0254321b-6a0d-4b22-abf1-fc77443e13f2 None admin] RBAC: Adding query filter params (role.id=43ca25467bc6484da3a50508efb79230, scope.project.id=5c653d5fa4304f298a295ce635efd5e6, user.id=4b347c9c89a64168b51c439b4a92d647) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:45.629595 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0254321b-6a0d-4b22-abf1-fc77443e13f2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:45.631115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0254321b-6a0d-4b22-abf1-fc77443e13f2 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:45.638070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 659/1287] 10.222.0.65 () {56 vars in 1374 bytes} [Wed Aug 24 03:30:45 2022] GET /identity/v3/role_assignments?user.id=4b347c9c89a64168b51c439b4a92d647&scope.project.id=5c653d5fa4304f298a295ce635efd5e6&role.id=43ca25467bc6484da3a50508efb79230 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:47.636719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbac1d7a-cfcb-4f4f-a050-15cdaaece22b None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:47.637297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbac1d7a-cfcb-4f4f-a050-15cdaaece22b None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:47.637297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbac1d7a-cfcb-4f4f-a050-15cdaaece22b None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:47.638640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 629/1288] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:30:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:30:47.645039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0561cf-e604-4a54-80e9-12c51fe14925 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:47.645524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0561cf-e604-4a54-80e9-12c51fe14925 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:47.645617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0561cf-e604-4a54-80e9-12c51fe14925 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:47.665513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-0a0561cf-e604-4a54-80e9-12c51fe14925 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:30:47.666772 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0a0561cf-e604-4a54-80e9-12c51fe14925 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:30:47.703280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:47.747148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 660/1289] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:30:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:30:48.018250 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbac1d7a-cfcb-4f4f-a050-15cdaaece22b None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:48.132817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbac1d7a-cfcb-4f4f-a050-15cdaaece22b None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:48.149069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:48.179693 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111e03bf-7648-45be-ab98-95c2a97b4db7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:48.180328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111e03bf-7648-45be-ab98-95c2a97b4db7 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:48.180601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111e03bf-7648-45be-ab98-95c2a97b4db7 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:48.180869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111e03bf-7648-45be-ab98-95c2a97b4db7 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:48.181452 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111e03bf-7648-45be-ab98-95c2a97b4db7 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:48.182846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111e03bf-7648-45be-ab98-95c2a97b4db7 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-111e03bf-7648-45be-ab98-95c2a97b4db7 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:48.192099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:48.193382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:48.194429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 630/1290] 10.222.0.65 () {56 vars in 1117 bytes} [Wed Aug 24 03:30:48 2022] GET /identity/v3/services/placement => generated 90 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:48.199839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0561cf-e604-4a54-80e9-12c51fe14925 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:48.303145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0561cf-e604-4a54-80e9-12c51fe14925 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:48.339890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f522b3-b80f-4109-833f-7e57f7c2f19a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:48.340542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f522b3-b80f-4109-833f-7e57f7c2f19a None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:48.340824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f522b3-b80f-4109-833f-7e57f7c2f19a None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:48.341094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f522b3-b80f-4109-833f-7e57f7c2f19a None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:48.341726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f522b3-b80f-4109-833f-7e57f7c2f19a None admin] RBAC: Adding query filter params (name=placement) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:48.342137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f522b3-b80f-4109-833f-7e57f7c2f19a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:48.343547 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f522b3-b80f-4109-833f-7e57f7c2f19a None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:48.348534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 661/1291] 10.222.0.65 () {56 vars in 1116 bytes} [Wed Aug 24 03:30:48 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:48.354472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111e03bf-7648-45be-ab98-95c2a97b4db7 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:48.465748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111e03bf-7648-45be-ab98-95c2a97b4db7 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:48.502289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525a7e79-f6ad-4629-9bb2-291ea627284f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:48.502899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525a7e79-f6ad-4629-9bb2-291ea627284f None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:48.503176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525a7e79-f6ad-4629-9bb2-291ea627284f None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:48.503447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525a7e79-f6ad-4629-9bb2-291ea627284f None admin] PATH_INFO: `/v3/services` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:48.504081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525a7e79-f6ad-4629-9bb2-291ea627284f None admin] RBAC: Adding query filter params (type=placement) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:48.504378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525a7e79-f6ad-4629-9bb2-291ea627284f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:48.505772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525a7e79-f6ad-4629-9bb2-291ea627284f None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:48.518219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 631/1292] 10.222.0.65 () {56 vars in 1116 bytes} [Wed Aug 24 03:30:48 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:30:50.460756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612486e1-03c9-4ad8-a7d8-c3a81978bcc0 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:50.461300 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612486e1-03c9-4ad8-a7d8-c3a81978bcc0 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:50.461300 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612486e1-03c9-4ad8-a7d8-c3a81978bcc0 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:50.462676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 662/1293] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:30:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:30:50.468394 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31edae61-0b99-4d52-bf2f-8c3adb72a83d None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:50.470956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31edae61-0b99-4d52-bf2f-8c3adb72a83d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:50.470956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31edae61-0b99-4d52-bf2f-8c3adb72a83d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:50.487649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-31edae61-0b99-4d52-bf2f-8c3adb72a83d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:30:50.488860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-31edae61-0b99-4d52-bf2f-8c3adb72a83d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:30:50.520117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:50.559779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 632/1294] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:30:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 24 03:30:50.838170 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612486e1-03c9-4ad8-a7d8-c3a81978bcc0 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:50.870143 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:50.952949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612486e1-03c9-4ad8-a7d8-c3a81978bcc0 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:50.991153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c41a63-c1bf-4754-85e7-3fa32ce55395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:50.991800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c41a63-c1bf-4754-85e7-3fa32ce55395 None admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:50.992083 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c41a63-c1bf-4754-85e7-3fa32ce55395 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:50.992485 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c41a63-c1bf-4754-85e7-3fa32ce55395 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:50.993485 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c41a63-c1bf-4754-85e7-3fa32ce55395 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:50.994933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c41a63-c1bf-4754-85e7-3fa32ce55395 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:51.232949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 663/1295] 10.222.0.65 () {60 vars in 1141 bytes} [Wed Aug 24 03:30:50 2022] POST /identity/v3/services => generated 246 bytes in 395 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:30:53.184629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984a2e4a-231f-4129-9d3a-f67b364062e2 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:53.185164 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984a2e4a-231f-4129-9d3a-f67b364062e2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:53.185164 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984a2e4a-231f-4129-9d3a-f67b364062e2 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:53.186544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 633/1296] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:30:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:30:53.192998 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1f1002-5148-49f9-bd64-f1d9eb9629df None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:53.193439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1f1002-5148-49f9-bd64-f1d9eb9629df None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:53.193538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1f1002-5148-49f9-bd64-f1d9eb9629df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:53.212607 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5e1f1002-5148-49f9-bd64-f1d9eb9629df None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:30:53.213926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5e1f1002-5148-49f9-bd64-f1d9eb9629df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:30:53.294812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 664/1297] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:30:53 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:30:53.569580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984a2e4a-231f-4129-9d3a-f67b364062e2 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:53.677430 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984a2e4a-231f-4129-9d3a-f67b364062e2 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:53.712988 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0c35ac-d3a3-4c7a-b6e0-49f808b50e88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:53.713616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0c35ac-d3a3-4c7a-b6e0-49f808b50e88 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:53.713891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0c35ac-d3a3-4c7a-b6e0-49f808b50e88 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:53.714179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0c35ac-d3a3-4c7a-b6e0-49f808b50e88 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:53.714778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0c35ac-d3a3-4c7a-b6e0-49f808b50e88 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:53.716171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0c35ac-d3a3-4c7a-b6e0-49f808b50e88 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-9c0c35ac-d3a3-4c7a-b6e0-49f808b50e88 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:53.725865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:53.728985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:53.729925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 634/1298] 10.222.0.65 () {56 vars in 1117 bytes} [Wed Aug 24 03:30:53 2022] GET /identity/v3/services/placement => generated 90 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:53.733300 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1f1002-5148-49f9-bd64-f1d9eb9629df None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:53.755811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:53.839387 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1f1002-5148-49f9-bd64-f1d9eb9629df None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:53.874690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6f0c96-4ad8-496b-9fca-7bef834ca46f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:53.875295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6f0c96-4ad8-496b-9fca-7bef834ca46f None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:53.876487 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6f0c96-4ad8-496b-9fca-7bef834ca46f None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:53.876911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6f0c96-4ad8-496b-9fca-7bef834ca46f None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:53.877546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6f0c96-4ad8-496b-9fca-7bef834ca46f None admin] RBAC: Adding query filter params (name=placement) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:53.877842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6f0c96-4ad8-496b-9fca-7bef834ca46f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:53.879235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6f0c96-4ad8-496b-9fca-7bef834ca46f None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:53.884745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 665/1299] 10.222.0.65 () {56 vars in 1116 bytes} [Wed Aug 24 03:30:53 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:53.891693 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0c35ac-d3a3-4c7a-b6e0-49f808b50e88 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:53.925390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:53.995423 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0c35ac-d3a3-4c7a-b6e0-49f808b50e88 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:54.029402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c9bffd-e2fa-4583-9927-4c6367691dfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:54.030005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c9bffd-e2fa-4583-9927-4c6367691dfb None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:54.030314 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c9bffd-e2fa-4583-9927-4c6367691dfb None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:54.030580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c9bffd-e2fa-4583-9927-4c6367691dfb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:54.031267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c9bffd-e2fa-4583-9927-4c6367691dfb None admin] RBAC: Adding query filter params (interface=public, service_id=736e2a7cc2e345bfa68dc649dde4f6ab, region_id=RegionOne) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:54.031590 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c9bffd-e2fa-4583-9927-4c6367691dfb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:54.033054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c9bffd-e2fa-4583-9927-4c6367691dfb None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:54.038402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 635/1300] 10.222.0.65 () {56 vars in 1251 bytes} [Wed Aug 24 03:30:53 2022] GET /identity/v3/endpoints?service_id=736e2a7cc2e345bfa68dc649dde4f6ab&interface=public®ion_id=RegionOne => generated 194 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:30:54.044579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6f0c96-4ad8-496b-9fca-7bef834ca46f None admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:54.151283 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6f0c96-4ad8-496b-9fca-7bef834ca46f None admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:54.185460 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c984e2d-037a-4051-a511-b276026cb6a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:54.186067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c984e2d-037a-4051-a511-b276026cb6a8 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:54.186375 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c984e2d-037a-4051-a511-b276026cb6a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:54.186649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c984e2d-037a-4051-a511-b276026cb6a8 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:54.187256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c984e2d-037a-4051-a511-b276026cb6a8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:54.189192 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c984e2d-037a-4051-a511-b276026cb6a8 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:54.197601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 666/1301] 10.222.0.65 () {56 vars in 1087 bytes} [Wed Aug 24 03:30:54 2022] GET /identity/v3/services => generated 2191 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:30:54.547891 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:56.170428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9119ac7a-2418-44f7-81c1-29ac23ae9f62 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:56.170971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9119ac7a-2418-44f7-81c1-29ac23ae9f62 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:56.170971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9119ac7a-2418-44f7-81c1-29ac23ae9f62 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:56.173080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 636/1302] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:30:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:30:56.178580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c2fd96-10ef-4442-9128-04edb7aec80b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:56.181365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c2fd96-10ef-4442-9128-04edb7aec80b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:56.181365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c2fd96-10ef-4442-9128-04edb7aec80b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:56.198327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-f7c2fd96-10ef-4442-9128-04edb7aec80b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:30:56.199539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f7c2fd96-10ef-4442-9128-04edb7aec80b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:30:56.269735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 667/1303] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:30:56 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 24 03:30:56.543885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9119ac7a-2418-44f7-81c1-29ac23ae9f62 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:56.653806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:56.660955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9119ac7a-2418-44f7-81c1-29ac23ae9f62 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:56.699190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37739bdd-27af-4bae-ace4-f8abb50f5f33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:56.700182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37739bdd-27af-4bae-ace4-f8abb50f5f33 None admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:56.700614 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37739bdd-27af-4bae-ace4-f8abb50f5f33 None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:56.701050 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37739bdd-27af-4bae-ace4-f8abb50f5f33 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:56.702088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37739bdd-27af-4bae-ace4-f8abb50f5f33 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:56.703393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37739bdd-27af-4bae-ace4-f8abb50f5f33 None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application [None req-37739bdd-27af-4bae-ace4-f8abb50f5f33 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 24 03:30:56.714514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 24 03:30:56.715609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.get_or_create( Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application with Lock( Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self._enter() Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application return self.creator() Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application created_value = creator( Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: ERROR keystone.server.flask.application Aug 24 03:30:56.716679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 637/1304] 10.222.0.65 () {56 vars in 1117 bytes} [Wed Aug 24 03:30:56 2022] GET /identity/v3/services/placement => generated 90 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 24 03:30:56.724344 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c2fd96-10ef-4442-9128-04edb7aec80b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:56.806911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:30:56.831054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c2fd96-10ef-4442-9128-04edb7aec80b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:56.867201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599d6050-ba1b-4536-8005-276b96a80262 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:56.870377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599d6050-ba1b-4536-8005-276b96a80262 None admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:56.870377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599d6050-ba1b-4536-8005-276b96a80262 None admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:56.870377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599d6050-ba1b-4536-8005-276b96a80262 None admin] PATH_INFO: `/v3/services` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:56.870377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599d6050-ba1b-4536-8005-276b96a80262 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:30:56.870377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599d6050-ba1b-4536-8005-276b96a80262 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:56.870802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599d6050-ba1b-4536-8005-276b96a80262 None admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:56.875868 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 668/1305] 10.222.0.65 () {56 vars in 1116 bytes} [Wed Aug 24 03:30:56 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:30:56.882310 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37739bdd-27af-4bae-ace4-f8abb50f5f33 None admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:30:56.986750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37739bdd-27af-4bae-ace4-f8abb50f5f33 None admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:30:57.021294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7542d8d6-fe09-4ae6-8936-35845d87729d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:30:57.021897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7542d8d6-fe09-4ae6-8936-35845d87729d None admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:30:57.022204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7542d8d6-fe09-4ae6-8936-35845d87729d None admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:30:57.022479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7542d8d6-fe09-4ae6-8936-35845d87729d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:30:57.023187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7542d8d6-fe09-4ae6-8936-35845d87729d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:30:57.024532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7542d8d6-fe09-4ae6-8936-35845d87729d None admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:30:57.134187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 638/1306] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:30:56 2022] POST /identity/v3/endpoints => generated 331 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:31:06.872978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=30 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:31:07.114135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:36.384326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269e0b34-2f8b-401f-80ca-f769f7282726 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:36.384917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269e0b34-2f8b-401f-80ca-f769f7282726 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:36.384917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269e0b34-2f8b-401f-80ca-f769f7282726 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:36.386257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 669/1307] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:39:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:39:36.392227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd561e10-d35d-4d55-b996-1e5e923c179e None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:36.393425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd561e10-d35d-4d55-b996-1e5e923c179e None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:36.393919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd561e10-d35d-4d55-b996-1e5e923c179e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:36.508695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-dd561e10-d35d-4d55-b996-1e5e923c179e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:39:36.510351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-dd561e10-d35d-4d55-b996-1e5e923c179e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:36.658302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 639/1308] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:39:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:39:36.970872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f366913e-fc08-40a4-98b9-118d36daeb5e None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:36.971511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f366913e-fc08-40a4-98b9-118d36daeb5e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:36.972209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f366913e-fc08-40a4-98b9-118d36daeb5e None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:36.974867 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 670/1309] 10.222.0.65 () {54 vars in 965 bytes} [Wed Aug 24 03:39:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:39:36.982425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52283ae5-3167-4f85-b1ba-46b18477c1c2 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:36.983068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52283ae5-3167-4f85-b1ba-46b18477c1c2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:36.983579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52283ae5-3167-4f85-b1ba-46b18477c1c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:37.095566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-52283ae5-3167-4f85-b1ba-46b18477c1c2 None None] MFA Rules not processed for user `64f23f0c8abc406b9700361723623387`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:39:37.096810 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-52283ae5-3167-4f85-b1ba-46b18477c1c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:37.123298 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:37.209421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 640/1310] 10.222.0.65 () {58 vars in 1062 bytes} [Wed Aug 24 03:39:36 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:39:37.218528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f366913e-fc08-40a4-98b9-118d36daeb5e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:39:37.220388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f366913e-fc08-40a4-98b9-118d36daeb5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:37.354245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f366913e-fc08-40a4-98b9-118d36daeb5e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:39:37.436403 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73faf6e6-9835-49b0-872d-1c8a6e547a69 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:39:37.437708 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73faf6e6-9835-49b0-872d-1c8a6e547a69 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:37.437990 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73faf6e6-9835-49b0-872d-1c8a6e547a69 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:37.438315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73faf6e6-9835-49b0-872d-1c8a6e547a69 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:37.439982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-73faf6e6-9835-49b0-872d-1c8a6e547a69 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:37.502544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:37.562738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73faf6e6-9835-49b0-872d-1c8a6e547a69 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:39:37.564550 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73faf6e6-9835-49b0-872d-1c8a6e547a69 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:39:37.585584 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 671/1311] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:39:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:39:39.842793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1535c85-b3ad-4fe4-bbb6-f6d5c13614c1 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:39.843301 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1535c85-b3ad-4fe4-bbb6-f6d5c13614c1 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:39.843381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1535c85-b3ad-4fe4-bbb6-f6d5c13614c1 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:39.846524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 641/1312] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:39:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:39:39.850275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c154d7b7-62a1-42cd-b567-c8e8568f0477 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:39.850832 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c154d7b7-62a1-42cd-b567-c8e8568f0477 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:39.850914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c154d7b7-62a1-42cd-b567-c8e8568f0477 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:39.935159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c154d7b7-62a1-42cd-b567-c8e8568f0477 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:39:39.936564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c154d7b7-62a1-42cd-b567-c8e8568f0477 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:40.041563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 672/1313] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:39:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:39:40.332503 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1535c85-b3ad-4fe4-bbb6-f6d5c13614c1 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:39:40.334216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b1535c85-b3ad-4fe4-bbb6-f6d5c13614c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:40.451558 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1535c85-b3ad-4fe4-bbb6-f6d5c13614c1 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:39:40.531975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:40.534733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f66399b-b413-4217-a1bc-eb774b643fcc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:39:40.535350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66399b-b413-4217-a1bc-eb774b643fcc service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:40.535628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66399b-b413-4217-a1bc-eb774b643fcc service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:40.535899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66399b-b413-4217-a1bc-eb774b643fcc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:40.537532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0f66399b-b413-4217-a1bc-eb774b643fcc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:40.664066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f66399b-b413-4217-a1bc-eb774b643fcc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:39:40.665515 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f66399b-b413-4217-a1bc-eb774b643fcc service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:39:40.684066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 642/1314] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:39:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:39:43.179388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e61a960-7849-40e1-b827-890ffe1b8a1e None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:43.180037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e61a960-7849-40e1-b827-890ffe1b8a1e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:43.180037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e61a960-7849-40e1-b827-890ffe1b8a1e None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:43.181234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 673/1315] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:39:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:39:43.186700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a812717-770d-48c5-ba18-2c9d22a4ba03 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:43.187111 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a812717-770d-48c5-ba18-2c9d22a4ba03 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:43.187497 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a812717-770d-48c5-ba18-2c9d22a4ba03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:43.263435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-0a812717-770d-48c5-ba18-2c9d22a4ba03 None None] MFA Rules not processed for user `18ccdd66160a423494d2172fb012e524`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:39:43.264666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0a812717-770d-48c5-ba18-2c9d22a4ba03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:43.320409 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:43.366516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 643/1316] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:39:43 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:39:43.674194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a8e14b-ca44-4832-901c-fcc667efa3cc None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:43.674492 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a8e14b-ca44-4832-901c-fcc667efa3cc None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:43.674765 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a8e14b-ca44-4832-901c-fcc667efa3cc None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:43.676096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 674/1317] 10.222.0.65 () {54 vars in 965 bytes} [Wed Aug 24 03:39:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:39:43.682984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943aa8c6-c86d-45de-a7c2-5f90785a549f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:43.683292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943aa8c6-c86d-45de-a7c2-5f90785a549f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:43.683568 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943aa8c6-c86d-45de-a7c2-5f90785a549f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:43.762270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-943aa8c6-c86d-45de-a7c2-5f90785a549f None None] MFA Rules not processed for user `64f23f0c8abc406b9700361723623387`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:39:43.763554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-943aa8c6-c86d-45de-a7c2-5f90785a549f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:43.841696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 644/1318] 10.222.0.65 () {58 vars in 1062 bytes} [Wed Aug 24 03:39:43 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:39:43.851245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a8e14b-ca44-4832-901c-fcc667efa3cc None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:39:43.852776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-55a8e14b-ca44-4832-901c-fcc667efa3cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:43.862668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:43.960690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a8e14b-ca44-4832-901c-fcc667efa3cc None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:39:44.035617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1af3543-f78f-480b-b9bd-e6e2606b7ec6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:39:44.036225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1af3543-f78f-480b-b9bd-e6e2606b7ec6 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:44.036506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1af3543-f78f-480b-b9bd-e6e2606b7ec6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:44.036782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1af3543-f78f-480b-b9bd-e6e2606b7ec6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:44.039032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c1af3543-f78f-480b-b9bd-e6e2606b7ec6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:44.101696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:44.149715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1af3543-f78f-480b-b9bd-e6e2606b7ec6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:39:44.151190 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1af3543-f78f-480b-b9bd-e6e2606b7ec6 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:39:44.170033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 675/1319] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:39:43 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:39:47.312308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b4261d-3782-456b-8314-8c6aa9bf3a02 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:47.312987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b4261d-3782-456b-8314-8c6aa9bf3a02 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:47.312987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b4261d-3782-456b-8314-8c6aa9bf3a02 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:47.314230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 645/1320] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:39:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:39:47.320418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c0a9ff-8caf-45b5-a1bc-c436bcf57779 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:47.320896 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c0a9ff-8caf-45b5-a1bc-c436bcf57779 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:47.320986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c0a9ff-8caf-45b5-a1bc-c436bcf57779 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:47.389963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-21c0a9ff-8caf-45b5-a1bc-c436bcf57779 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:39:47.391221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-21c0a9ff-8caf-45b5-a1bc-c436bcf57779 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:47.503124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 676/1321] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:39:47 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:39:47.786735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b4261d-3782-456b-8314-8c6aa9bf3a02 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:39:47.788298 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c6b4261d-3782-456b-8314-8c6aa9bf3a02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:47.842596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:47.906864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b4261d-3782-456b-8314-8c6aa9bf3a02 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:39:47.990208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877bf722-5eb8-463e-808e-cf9953f078b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:39:47.990804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877bf722-5eb8-463e-808e-cf9953f078b1 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:47.991081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877bf722-5eb8-463e-808e-cf9953f078b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:47.991540 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877bf722-5eb8-463e-808e-cf9953f078b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:47.993372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-877bf722-5eb8-463e-808e-cf9953f078b1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:48.092788 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:48.106245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877bf722-5eb8-463e-808e-cf9953f078b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:39:48.107687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877bf722-5eb8-463e-808e-cf9953f078b1 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:39:48.127021 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 646/1322] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:39:47 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:39:51.202450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baea48a8-18b6-4408-b4dd-3fa6ec0a1642 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:51.203019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baea48a8-18b6-4408-b4dd-3fa6ec0a1642 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:51.203019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baea48a8-18b6-4408-b4dd-3fa6ec0a1642 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:51.204323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 677/1323] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:39:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:39:51.209574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03796eb-f1d0-40b6-b308-e937f0aad563 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:51.210195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03796eb-f1d0-40b6-b308-e937f0aad563 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:51.210195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03796eb-f1d0-40b6-b308-e937f0aad563 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:51.292745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d03796eb-f1d0-40b6-b308-e937f0aad563 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:39:51.294049 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d03796eb-f1d0-40b6-b308-e937f0aad563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:51.403655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 647/1324] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:39:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:39:51.695592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baea48a8-18b6-4408-b4dd-3fa6ec0a1642 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:39:51.697159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-baea48a8-18b6-4408-b4dd-3fa6ec0a1642 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:51.719967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:51.819898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baea48a8-18b6-4408-b4dd-3fa6ec0a1642 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:39:51.902871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b65b8a1-0f49-4234-9651-95d5997b6442 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:39:51.903480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b65b8a1-0f49-4234-9651-95d5997b6442 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:51.903762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b65b8a1-0f49-4234-9651-95d5997b6442 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:51.904030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b65b8a1-0f49-4234-9651-95d5997b6442 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:51.905623 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8b65b8a1-0f49-4234-9651-95d5997b6442 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:51.983925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:52.024016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b65b8a1-0f49-4234-9651-95d5997b6442 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:39:52.025387 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b65b8a1-0f49-4234-9651-95d5997b6442 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:39:52.044646 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 678/1325] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:39:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:39:54.075187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e30bd35-37f8-421e-81c0-c841c5335b63 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:54.075807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e30bd35-37f8-421e-81c0-c841c5335b63 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:54.075807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e30bd35-37f8-421e-81c0-c841c5335b63 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:54.077094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 648/1326] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:39:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:39:54.082783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4848fc1b-89e6-41a7-8643-0c4de751fdaf None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:54.083382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4848fc1b-89e6-41a7-8643-0c4de751fdaf None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:54.083382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4848fc1b-89e6-41a7-8643-0c4de751fdaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:54.159882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-4848fc1b-89e6-41a7-8643-0c4de751fdaf None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:39:54.161003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4848fc1b-89e6-41a7-8643-0c4de751fdaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:54.274450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 679/1327] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:39:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:39:54.566709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e30bd35-37f8-421e-81c0-c841c5335b63 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:39:54.568314 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-2e30bd35-37f8-421e-81c0-c841c5335b63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:54.642421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:54.690940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e30bd35-37f8-421e-81c0-c841c5335b63 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:39:54.778577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51b3069-f5eb-4590-9463-e9bb9d41d1b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:39:54.779184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51b3069-f5eb-4590-9463-e9bb9d41d1b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:54.779460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51b3069-f5eb-4590-9463-e9bb9d41d1b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:54.779728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51b3069-f5eb-4590-9463-e9bb9d41d1b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:54.781506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d51b3069-f5eb-4590-9463-e9bb9d41d1b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:54.906911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:54.911702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51b3069-f5eb-4590-9463-e9bb9d41d1b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:39:54.911702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51b3069-f5eb-4590-9463-e9bb9d41d1b4 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:39:54.930992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 649/1328] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:39:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:39:56.949408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63ce4a5-582f-415f-b742-2d38ca17f7fe None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:56.949966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63ce4a5-582f-415f-b742-2d38ca17f7fe None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:56.949966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63ce4a5-582f-415f-b742-2d38ca17f7fe None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:56.951345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 680/1329] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:39:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:39:56.957477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328832fc-5a7e-49c8-8ba1-a07277444a10 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:56.957986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328832fc-5a7e-49c8-8ba1-a07277444a10 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:56.957986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328832fc-5a7e-49c8-8ba1-a07277444a10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:57.035988 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-328832fc-5a7e-49c8-8ba1-a07277444a10 None None] MFA Rules not processed for user `18ccdd66160a423494d2172fb012e524`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:39:57.036987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-328832fc-5a7e-49c8-8ba1-a07277444a10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:57.146694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 650/1330] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:39:56 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:39:57.435907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63ce4a5-582f-415f-b742-2d38ca17f7fe None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:39:57.437462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a63ce4a5-582f-415f-b742-2d38ca17f7fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:57.473224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:57.566709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63ce4a5-582f-415f-b742-2d38ca17f7fe None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:39:57.659955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13920b9b-19a1-4dd2-bcd6-85d6b77ce74a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:39:57.660568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13920b9b-19a1-4dd2-bcd6-85d6b77ce74a service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:39:57.660846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13920b9b-19a1-4dd2-bcd6-85d6b77ce74a service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:39:57.661118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13920b9b-19a1-4dd2-bcd6-85d6b77ce74a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:39:57.663439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-13920b9b-19a1-4dd2-bcd6-85d6b77ce74a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:39:57.745335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:39:57.781639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13920b9b-19a1-4dd2-bcd6-85d6b77ce74a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:39:57.783082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13920b9b-19a1-4dd2-bcd6-85d6b77ce74a service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:39:57.802736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 681/1331] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:39:57 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:00.389705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294c3bbc-e943-4d58-9f01-53115c6e3eab None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:00.390419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294c3bbc-e943-4d58-9f01-53115c6e3eab None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:00.390419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294c3bbc-e943-4d58-9f01-53115c6e3eab None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:00.391705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 651/1332] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:40:00.398174 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740dc713-ddb7-44a5-9c37-0c160be6861f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:00.398665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740dc713-ddb7-44a5-9c37-0c160be6861f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:00.398841 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740dc713-ddb7-44a5-9c37-0c160be6861f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:00.467008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-740dc713-ddb7-44a5-9c37-0c160be6861f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:00.468388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-740dc713-ddb7-44a5-9c37-0c160be6861f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:00.569555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 682/1333] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:00.851981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294c3bbc-e943-4d58-9f01-53115c6e3eab None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:00.853522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-294c3bbc-e943-4d58-9f01-53115c6e3eab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:00.925471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:00.960810 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294c3bbc-e943-4d58-9f01-53115c6e3eab None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:01.036760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f53891c-be27-443d-858d-34d0bd3091b9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:01.037365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53891c-be27-443d-858d-34d0bd3091b9 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:01.038177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53891c-be27-443d-858d-34d0bd3091b9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:01.038453 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f53891c-be27-443d-858d-34d0bd3091b9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:01.040079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6f53891c-be27-443d-858d-34d0bd3091b9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:01.150882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f53891c-be27-443d-858d-34d0bd3091b9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:01.152246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f53891c-be27-443d-858d-34d0bd3091b9 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:01.162209 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:01.171094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 652/1334] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:03.223648 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0085b5c7-0d71-4cf4-8668-51e99ea2a001 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:03.224199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0085b5c7-0d71-4cf4-8668-51e99ea2a001 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:03.224608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0085b5c7-0d71-4cf4-8668-51e99ea2a001 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:03.227903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 683/1335] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:03 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:03.233669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927df3bb-07d2-4bd7-89b8-8c65734cef73 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:03.234109 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927df3bb-07d2-4bd7-89b8-8c65734cef73 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:03.234225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927df3bb-07d2-4bd7-89b8-8c65734cef73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:03.302362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-927df3bb-07d2-4bd7-89b8-8c65734cef73 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:03.303646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-927df3bb-07d2-4bd7-89b8-8c65734cef73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:03.408471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 653/1336] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:40:03.702575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0085b5c7-0d71-4cf4-8668-51e99ea2a001 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:03.705363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0085b5c7-0d71-4cf4-8668-51e99ea2a001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:03.748982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:03.816413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0085b5c7-0d71-4cf4-8668-51e99ea2a001 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:03.894070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055169f6-03e5-4d8a-a377-f23f105ed705 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:03.894834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055169f6-03e5-4d8a-a377-f23f105ed705 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:03.895237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055169f6-03e5-4d8a-a377-f23f105ed705 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:03.896090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055169f6-03e5-4d8a-a377-f23f105ed705 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:03.897908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-055169f6-03e5-4d8a-a377-f23f105ed705 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:03.987138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:04.014725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055169f6-03e5-4d8a-a377-f23f105ed705 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:04.016197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055169f6-03e5-4d8a-a377-f23f105ed705 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:04.035471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 684/1337] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:06.592015 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d4bd54-041d-4ad3-8a06-403574997d8f None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:06.592566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d4bd54-041d-4ad3-8a06-403574997d8f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:06.592566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d4bd54-041d-4ad3-8a06-403574997d8f None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:06.593968 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 654/1338] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:06.599411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dc7fcc-bece-41c0-a17a-5ee2265ef385 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:06.599925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dc7fcc-bece-41c0-a17a-5ee2265ef385 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:06.600585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dc7fcc-bece-41c0-a17a-5ee2265ef385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:06.670109 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-f7dc7fcc-bece-41c0-a17a-5ee2265ef385 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:06.671280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f7dc7fcc-bece-41c0-a17a-5ee2265ef385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:06.782258 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 685/1339] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:06 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:07.092108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d4bd54-041d-4ad3-8a06-403574997d8f None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:07.093647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-03d4bd54-041d-4ad3-8a06-403574997d8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:07.169407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:07.197564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d4bd54-041d-4ad3-8a06-403574997d8f None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:07.271213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46c03c2-89c0-4495-91a9-78918a72d614 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:07.271740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46c03c2-89c0-4495-91a9-78918a72d614 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:07.272119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46c03c2-89c0-4495-91a9-78918a72d614 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:07.272377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46c03c2-89c0-4495-91a9-78918a72d614 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:07.273973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d46c03c2-89c0-4495-91a9-78918a72d614 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:07.383585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46c03c2-89c0-4495-91a9-78918a72d614 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:07.385259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46c03c2-89c0-4495-91a9-78918a72d614 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:07.403744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 655/1340] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:07 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:11.844382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb41fc3a-aabf-43e6-b1b6-01ed43323808 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:11.844977 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb41fc3a-aabf-43e6-b1b6-01ed43323808 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:11.844977 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb41fc3a-aabf-43e6-b1b6-01ed43323808 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:11.846477 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 686/1341] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:11.851894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ada0495-e78c-4db1-8620-596b529d3f2e None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:11.852400 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ada0495-e78c-4db1-8620-596b529d3f2e None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:11.852589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ada0495-e78c-4db1-8620-596b529d3f2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:11.857982 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:11.923775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-9ada0495-e78c-4db1-8620-596b529d3f2e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:11.924992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-9ada0495-e78c-4db1-8620-596b529d3f2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:12.031206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 656/1342] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:12.324524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb41fc3a-aabf-43e6-b1b6-01ed43323808 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:12.326197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-bb41fc3a-aabf-43e6-b1b6-01ed43323808 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:12.380684 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:12.440667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb41fc3a-aabf-43e6-b1b6-01ed43323808 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:12.527934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f4261c-6840-4172-94fc-193d1abd0498 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:12.528530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f4261c-6840-4172-94fc-193d1abd0498 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:12.528811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f4261c-6840-4172-94fc-193d1abd0498 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:12.529068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f4261c-6840-4172-94fc-193d1abd0498 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:12.530717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-00f4261c-6840-4172-94fc-193d1abd0498 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:12.624824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:12.641516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f4261c-6840-4172-94fc-193d1abd0498 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:12.643078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f4261c-6840-4172-94fc-193d1abd0498 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:12.661804 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 687/1343] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:15.595755 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cd2df3-61d4-4c74-aab6-3c1d2eb94419 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:15.596288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cd2df3-61d4-4c74-aab6-3c1d2eb94419 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:15.596288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cd2df3-61d4-4c74-aab6-3c1d2eb94419 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:15.597644 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 657/1344] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:15.603830 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a09a3f-04b2-4793-b7a4-46dba713e365 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:15.604264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a09a3f-04b2-4793-b7a4-46dba713e365 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:15.604456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a09a3f-04b2-4793-b7a4-46dba713e365 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:15.671801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d4a09a3f-04b2-4793-b7a4-46dba713e365 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:15.673038 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d4a09a3f-04b2-4793-b7a4-46dba713e365 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:15.774575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 688/1345] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:15 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:16.061778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cd2df3-61d4-4c74-aab6-3c1d2eb94419 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:16.063381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-e6cd2df3-61d4-4c74-aab6-3c1d2eb94419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:16.152660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:16.169355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cd2df3-61d4-4c74-aab6-3c1d2eb94419 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:16.248577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d919b4cb-d24a-47f7-a96d-684bff796ab7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:16.249243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d919b4cb-d24a-47f7-a96d-684bff796ab7 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:16.249550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d919b4cb-d24a-47f7-a96d-684bff796ab7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:16.250027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d919b4cb-d24a-47f7-a96d-684bff796ab7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:16.251442 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d919b4cb-d24a-47f7-a96d-684bff796ab7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:16.372110 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d919b4cb-d24a-47f7-a96d-684bff796ab7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:16.373559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d919b4cb-d24a-47f7-a96d-684bff796ab7 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:16.392883 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 658/1346] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:16 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:20.660603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4c4e84-2c98-403a-863b-f116c126102d None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:20.661153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4c4e84-2c98-403a-863b-f116c126102d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:20.661153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4c4e84-2c98-403a-863b-f116c126102d None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:20.662684 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 689/1347] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:20.668623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ca80f4-db2b-402e-ad11-ca32dd74e581 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:20.669142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ca80f4-db2b-402e-ad11-ca32dd74e581 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:20.669252 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ca80f4-db2b-402e-ad11-ca32dd74e581 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:20.686263 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:20.748330 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f5ca80f4-db2b-402e-ad11-ca32dd74e581 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:20.749574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f5ca80f4-db2b-402e-ad11-ca32dd74e581 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:20.858627 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 659/1348] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:40:21.143756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4c4e84-2c98-403a-863b-f116c126102d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:21.145754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-dc4c4e84-2c98-403a-863b-f116c126102d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:21.216646 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:21.275558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4c4e84-2c98-403a-863b-f116c126102d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:21.350110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed16e544-92a3-409b-b8d8-b593da5d7828 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:21.350833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed16e544-92a3-409b-b8d8-b593da5d7828 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:21.351114 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed16e544-92a3-409b-b8d8-b593da5d7828 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:21.351385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed16e544-92a3-409b-b8d8-b593da5d7828 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:21.352991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ed16e544-92a3-409b-b8d8-b593da5d7828 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:21.462133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:21.465799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed16e544-92a3-409b-b8d8-b593da5d7828 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:21.467302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed16e544-92a3-409b-b8d8-b593da5d7828 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:21.489997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 690/1349] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:26.636039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a344d-1200-4106-a927-ff404b82e6da None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:26.636584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a344d-1200-4106-a927-ff404b82e6da None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:26.636584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a344d-1200-4106-a927-ff404b82e6da None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:26.642199 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 660/1350] 10.222.0.65 () {54 vars in 921 bytes} [Wed Aug 24 03:40:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:26.649016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439efc09-cfd0-4f2f-9ddf-ba394d4ac710 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:26.649512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439efc09-cfd0-4f2f-9ddf-ba394d4ac710 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:26.649778 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439efc09-cfd0-4f2f-9ddf-ba394d4ac710 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:26.759252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-439efc09-cfd0-4f2f-9ddf-ba394d4ac710 None None] MFA Rules not processed for user `4b347c9c89a64168b51c439b4a92d647`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:26.760392 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-439efc09-cfd0-4f2f-9ddf-ba394d4ac710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:26.912415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 691/1351] 10.222.0.65 () {58 vars in 1018 bytes} [Wed Aug 24 03:40:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:30.856827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=59 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:31.202977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80da4b-27b1-4a22-993f-68932749cd47 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:31.203232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80da4b-27b1-4a22-993f-68932749cd47 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:31.203542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80da4b-27b1-4a22-993f-68932749cd47 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:31.205029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 661/1352] 10.222.0.65 () {54 vars in 921 bytes} [Wed Aug 24 03:40:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:31.210645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dcb988-7999-4891-b91d-a65f4fad5997 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:31.211110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dcb988-7999-4891-b91d-a65f4fad5997 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:31.211193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dcb988-7999-4891-b91d-a65f4fad5997 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:31.268407 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:31.293910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-25dcb988-7999-4891-b91d-a65f4fad5997 None None] MFA Rules not processed for user `4b347c9c89a64168b51c439b4a92d647`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:31.295245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-25dcb988-7999-4891-b91d-a65f4fad5997 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:31.417052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 692/1353] 10.222.0.65 () {58 vars in 1018 bytes} [Wed Aug 24 03:40:31 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:32.906214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b929c4b-8847-4510-b2c9-45566302205a None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:32.906784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b929c4b-8847-4510-b2c9-45566302205a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:32.906877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b929c4b-8847-4510-b2c9-45566302205a None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:32.908208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 662/1354] 10.222.0.65 () {54 vars in 921 bytes} [Wed Aug 24 03:40:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:32.914312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c049949-4b23-4d66-aa49-9b0fb1b35ac6 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:32.914729 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c049949-4b23-4d66-aa49-9b0fb1b35ac6 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:32.914989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c049949-4b23-4d66-aa49-9b0fb1b35ac6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:32.991020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1c049949-4b23-4d66-aa49-9b0fb1b35ac6 None None] MFA Rules not processed for user `4b347c9c89a64168b51c439b4a92d647`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:32.992220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1c049949-4b23-4d66-aa49-9b0fb1b35ac6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:33.034608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:33.105604 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 693/1355] 10.222.0.65 () {58 vars in 1018 bytes} [Wed Aug 24 03:40:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:40:36.435527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b23df4d-a4a3-42a5-ba7f-ea2ddcd90b34 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:36.436053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b23df4d-a4a3-42a5-ba7f-ea2ddcd90b34 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:36.436150 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b23df4d-a4a3-42a5-ba7f-ea2ddcd90b34 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:36.437443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 663/1356] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:36.442922 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4081cc77-cf8f-4968-884e-c2a01bdd7b2e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:36.444107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4081cc77-cf8f-4968-884e-c2a01bdd7b2e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:36.444107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4081cc77-cf8f-4968-884e-c2a01bdd7b2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:36.528761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-4081cc77-cf8f-4968-884e-c2a01bdd7b2e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:36.529955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4081cc77-cf8f-4968-884e-c2a01bdd7b2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:36.664332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 694/1357] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:36.976813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9a083d-cd31-4215-8426-efc9f640783d None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:36.977419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9a083d-cd31-4215-8426-efc9f640783d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:36.984304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9a083d-cd31-4215-8426-efc9f640783d None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:36.984304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 664/1358] 10.222.0.65 () {54 vars in 962 bytes} [Wed Aug 24 03:40:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:36.995392 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab34242-d597-45e9-a70c-14b3e031f6c1 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:36.995695 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab34242-d597-45e9-a70c-14b3e031f6c1 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:36.995976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab34242-d597-45e9-a70c-14b3e031f6c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:37.016013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:37.088663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-8ab34242-d597-45e9-a70c-14b3e031f6c1 None None] MFA Rules not processed for user `e5896018712d4b82a781365e478ee855`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:37.090186 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8ab34242-d597-45e9-a70c-14b3e031f6c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:37.210186 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 695/1359] 10.222.0.65 () {58 vars in 1059 bytes} [Wed Aug 24 03:40:36 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:37.217999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9a083d-cd31-4215-8426-efc9f640783d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:37.219641 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4e9a083d-cd31-4215-8426-efc9f640783d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:37.384652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9a083d-cd31-4215-8426-efc9f640783d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:37.467344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:37.513363 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec9884-59fb-4791-b53f-e9782af9b34f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:37.514563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec9884-59fb-4791-b53f-e9782af9b34f service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:37.515114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec9884-59fb-4791-b53f-e9782af9b34f service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:37.515784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec9884-59fb-4791-b53f-e9782af9b34f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:37.517657 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d4ec9884-59fb-4791-b53f-e9782af9b34f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:37.614547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ec9884-59fb-4791-b53f-e9782af9b34f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:37.616918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ec9884-59fb-4791-b53f-e9782af9b34f service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:37.636096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 665/1360] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:39.812195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2fe901-ffd0-4f70-ba3c-b792a18af6b6 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:39.812840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2fe901-ffd0-4f70-ba3c-b792a18af6b6 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:39.812840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2fe901-ffd0-4f70-ba3c-b792a18af6b6 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:39.818034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 696/1361] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:39 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:39.819833 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d5f85f-1aac-4dfa-8bca-06c0eb2c167f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:39.820322 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d5f85f-1aac-4dfa-8bca-06c0eb2c167f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:39.820699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d5f85f-1aac-4dfa-8bca-06c0eb2c167f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:39.896828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-43d5f85f-1aac-4dfa-8bca-06c0eb2c167f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:39.898227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-43d5f85f-1aac-4dfa-8bca-06c0eb2c167f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:39.940750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:40.002929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 666/1362] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:40.291504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2fe901-ffd0-4f70-ba3c-b792a18af6b6 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:40.293725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ee2fe901-ffd0-4f70-ba3c-b792a18af6b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:40.360717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:40.456154 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2fe901-ffd0-4f70-ba3c-b792a18af6b6 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:40.579773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638672b8-672e-4d72-8bd6-9f99e636d369 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:40.580481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638672b8-672e-4d72-8bd6-9f99e636d369 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:40.580868 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638672b8-672e-4d72-8bd6-9f99e636d369 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:40.581254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638672b8-672e-4d72-8bd6-9f99e636d369 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:40.582998 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-638672b8-672e-4d72-8bd6-9f99e636d369 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:40.608734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:40.685989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638672b8-672e-4d72-8bd6-9f99e636d369 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:40.689653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638672b8-672e-4d72-8bd6-9f99e636d369 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:40.712201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 697/1363] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:43.056530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6745f802-628e-47fe-8898-accb773703fa None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:43.057104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6745f802-628e-47fe-8898-accb773703fa None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:43.057104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6745f802-628e-47fe-8898-accb773703fa None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:43.058475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 667/1364] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:43.064085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68414ff1-b13b-46a5-a3e7-64cef868fae3 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:43.064518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68414ff1-b13b-46a5-a3e7-64cef868fae3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:43.064704 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68414ff1-b13b-46a5-a3e7-64cef868fae3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:43.142087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-68414ff1-b13b-46a5-a3e7-64cef868fae3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:43.143324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-68414ff1-b13b-46a5-a3e7-64cef868fae3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:43.268877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:43.272819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 698/1365] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:40:43.597554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3241f195-b6b3-4b85-ab07-90fc33bbe651 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:43.597956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3241f195-b6b3-4b85-ab07-90fc33bbe651 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:43.598114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3241f195-b6b3-4b85-ab07-90fc33bbe651 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:43.601220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 668/1366] 10.222.0.65 () {54 vars in 962 bytes} [Wed Aug 24 03:40:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:43.608815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dbe260-7bae-48bd-9261-37c66384865b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:43.609218 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dbe260-7bae-48bd-9261-37c66384865b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:43.609620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dbe260-7bae-48bd-9261-37c66384865b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:43.697047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-67dbe260-7bae-48bd-9261-37c66384865b None None] MFA Rules not processed for user `e5896018712d4b82a781365e478ee855`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:43.698473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-67dbe260-7bae-48bd-9261-37c66384865b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:43.839938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 699/1367] 10.222.0.65 () {58 vars in 1059 bytes} [Wed Aug 24 03:40:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:43.849686 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3241f195-b6b3-4b85-ab07-90fc33bbe651 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:43.851638 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3241f195-b6b3-4b85-ab07-90fc33bbe651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:44.036151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3241f195-b6b3-4b85-ab07-90fc33bbe651 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:44.062077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:44.164378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ba88b7-ce5b-43c8-91e9-ea2fc58a42a5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:44.164897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ba88b7-ce5b-43c8-91e9-ea2fc58a42a5 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:44.165169 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ba88b7-ce5b-43c8-91e9-ea2fc58a42a5 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:44.165432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ba88b7-ce5b-43c8-91e9-ea2fc58a42a5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:44.167057 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-e7ba88b7-ce5b-43c8-91e9-ea2fc58a42a5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:44.279879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ba88b7-ce5b-43c8-91e9-ea2fc58a42a5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:44.281480 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ba88b7-ce5b-43c8-91e9-ea2fc58a42a5 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:44.301316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 669/1368] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:40:44.835828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d24d1-983c-490a-8879-683daade02d3 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:44.836507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d24d1-983c-490a-8879-683daade02d3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:44.836507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d24d1-983c-490a-8879-683daade02d3 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:44.837569 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 700/1369] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:40:44.843450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb64bbdd-03af-4243-85a3-74782095c3fd None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:44.843881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb64bbdd-03af-4243-85a3-74782095c3fd None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:44.844378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb64bbdd-03af-4243-85a3-74782095c3fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:44.880441 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:44.919836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-fb64bbdd-03af-4243-85a3-74782095c3fd None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:44.921220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-fb64bbdd-03af-4243-85a3-74782095c3fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:45.025869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 670/1370] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:45.619439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31370613-d35f-4e53-93b7-0287f623f38c None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:45.619726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31370613-d35f-4e53-93b7-0287f623f38c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:45.620002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31370613-d35f-4e53-93b7-0287f623f38c None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:45.621692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 701/1371] 10.222.0.65 () {54 vars in 964 bytes} [Wed Aug 24 03:40:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:45.626736 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55b3eeb-1bab-4c8c-a256-20ba1b31f53a None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:45.627216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55b3eeb-1bab-4c8c-a256-20ba1b31f53a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:45.627312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55b3eeb-1bab-4c8c-a256-20ba1b31f53a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:45.716528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-c55b3eeb-1bab-4c8c-a256-20ba1b31f53a None None] MFA Rules not processed for user `dbfbfd64573141f88040692119ac5a1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:45.717825 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c55b3eeb-1bab-4c8c-a256-20ba1b31f53a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:45.771049 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:45.801565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 671/1372] 10.222.0.65 () {58 vars in 1061 bytes} [Wed Aug 24 03:40:45 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:45.810559 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31370613-d35f-4e53-93b7-0287f623f38c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:45.812127 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-31370613-d35f-4e53-93b7-0287f623f38c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:45.865376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:45.932394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31370613-d35f-4e53-93b7-0287f623f38c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:46.016958 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e95a67-6a60-46d9-9f6c-de6edf53114c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:46.017563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e95a67-6a60-46d9-9f6c-de6edf53114c service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:46.017842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e95a67-6a60-46d9-9f6c-de6edf53114c service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:46.018113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e95a67-6a60-46d9-9f6c-de6edf53114c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:46.019729 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c9e95a67-6a60-46d9-9f6c-de6edf53114c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:46.166538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:46.181488 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e95a67-6a60-46d9-9f6c-de6edf53114c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:46.181927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e95a67-6a60-46d9-9f6c-de6edf53114c service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:46.204378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 702/1373] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:40:46.589778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764a8b1c-ec0f-470f-819c-4b2e2d9faaec None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:46.589972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764a8b1c-ec0f-470f-819c-4b2e2d9faaec None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:46.590259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764a8b1c-ec0f-470f-819c-4b2e2d9faaec None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:46.591619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 672/1374] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:46.596822 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c0eb52-150a-40bb-ad8f-1170a1a54680 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:46.597177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c0eb52-150a-40bb-ad8f-1170a1a54680 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:46.597524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c0eb52-150a-40bb-ad8f-1170a1a54680 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:46.665372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-f9c0eb52-150a-40bb-ad8f-1170a1a54680 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:46.666748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f9c0eb52-150a-40bb-ad8f-1170a1a54680 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:46.774585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 703/1375] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:47.070800 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764a8b1c-ec0f-470f-819c-4b2e2d9faaec None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:47.072393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-764a8b1c-ec0f-470f-819c-4b2e2d9faaec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:47.229747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764a8b1c-ec0f-470f-819c-4b2e2d9faaec None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:47.283473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:47.348502 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd40886-d2ee-4908-b5b1-1e3e9c02d555 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:47.349250 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd40886-d2ee-4908-b5b1-1e3e9c02d555 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:47.349592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd40886-d2ee-4908-b5b1-1e3e9c02d555 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:47.349969 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd40886-d2ee-4908-b5b1-1e3e9c02d555 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:47.352429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-bdd40886-d2ee-4908-b5b1-1e3e9c02d555 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:47.451867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd40886-d2ee-4908-b5b1-1e3e9c02d555 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:47.453474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd40886-d2ee-4908-b5b1-1e3e9c02d555 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:47.472551 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 673/1376] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:49.524568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4144c464-a057-45cd-b86d-e0b973605588 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:49.525053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4144c464-a057-45cd-b86d-e0b973605588 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:49.525053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4144c464-a057-45cd-b86d-e0b973605588 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:49.526858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 704/1377] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:49.532528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4f6269-840d-4649-9045-46c1239cac32 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:49.533021 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4f6269-840d-4649-9045-46c1239cac32 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:49.533155 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4f6269-840d-4649-9045-46c1239cac32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:49.608932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-de4f6269-840d-4649-9045-46c1239cac32 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:49.610181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-de4f6269-840d-4649-9045-46c1239cac32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:49.617974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:49.713528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 674/1378] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:50.035387 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4144c464-a057-45cd-b86d-e0b973605588 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:50.039081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4144c464-a057-45cd-b86d-e0b973605588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:50.109650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:50.202778 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4144c464-a057-45cd-b86d-e0b973605588 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:50.327544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4a67b7-2fe9-42f2-bd71-df44472d8ec0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:50.328247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4a67b7-2fe9-42f2-bd71-df44472d8ec0 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:50.328624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4a67b7-2fe9-42f2-bd71-df44472d8ec0 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:50.328996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4a67b7-2fe9-42f2-bd71-df44472d8ec0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:50.330715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7d4a67b7-2fe9-42f2-bd71-df44472d8ec0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:50.351005 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:50.432256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4a67b7-2fe9-42f2-bd71-df44472d8ec0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:50.433995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4a67b7-2fe9-42f2-bd71-df44472d8ec0 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:50.454089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 705/1379] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:52.601822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f087ac-3596-4d43-a6d7-536e618ef53c None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:52.602451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f087ac-3596-4d43-a6d7-536e618ef53c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:52.602451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f087ac-3596-4d43-a6d7-536e618ef53c None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:52.603956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 675/1380] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:52.609449 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4797308-e6a9-4d03-87f8-782a09cf2dc3 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:52.610423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4797308-e6a9-4d03-87f8-782a09cf2dc3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:52.610979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4797308-e6a9-4d03-87f8-782a09cf2dc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:52.694883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e4797308-e6a9-4d03-87f8-782a09cf2dc3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:52.696911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e4797308-e6a9-4d03-87f8-782a09cf2dc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:52.805280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:52.812295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 706/1381] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:53.109437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f087ac-3596-4d43-a6d7-536e618ef53c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:53.111235 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-63f087ac-3596-4d43-a6d7-536e618ef53c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:53.262847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:53.274341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f087ac-3596-4d43-a6d7-536e618ef53c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:53.401830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf0011d-51fc-435c-abf5-064132b98df3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:53.402383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf0011d-51fc-435c-abf5-064132b98df3 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:53.402659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf0011d-51fc-435c-abf5-064132b98df3 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:53.402928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf0011d-51fc-435c-abf5-064132b98df3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:53.404581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ccf0011d-51fc-435c-abf5-064132b98df3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:53.512849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf0011d-51fc-435c-abf5-064132b98df3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:53.514522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf0011d-51fc-435c-abf5-064132b98df3 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:53.526133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:53.534938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 676/1382] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:55.887335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d35bfd6-270a-4dc7-9e9c-a849059437cb None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:55.888165 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d35bfd6-270a-4dc7-9e9c-a849059437cb None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:55.888165 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d35bfd6-270a-4dc7-9e9c-a849059437cb None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:55.889440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 707/1383] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:40:55.895303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa1f10-4789-4da8-abf8-a8af4d9921c5 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:55.896427 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa1f10-4789-4da8-abf8-a8af4d9921c5 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:55.896584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa1f10-4789-4da8-abf8-a8af4d9921c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:55.969084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-58fa1f10-4789-4da8-abf8-a8af4d9921c5 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:55.970829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-58fa1f10-4789-4da8-abf8-a8af4d9921c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:56.078767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 677/1384] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:56.367973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d35bfd6-270a-4dc7-9e9c-a849059437cb None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:56.370257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1d35bfd6-270a-4dc7-9e9c-a849059437cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:56.536763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d35bfd6-270a-4dc7-9e9c-a849059437cb None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:56.622063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:56.673278 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f628f5d6-f950-4417-8bfd-765daf7ac71e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:56.673888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f628f5d6-f950-4417-8bfd-765daf7ac71e service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:56.674176 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f628f5d6-f950-4417-8bfd-765daf7ac71e service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:56.674470 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f628f5d6-f950-4417-8bfd-765daf7ac71e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:56.676086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f628f5d6-f950-4417-8bfd-765daf7ac71e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:56.780762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f628f5d6-f950-4417-8bfd-765daf7ac71e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:56.782864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f628f5d6-f950-4417-8bfd-765daf7ac71e service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:56.803494 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 708/1385] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:40:58.723236 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354302d-5312-45d8-80ab-208da41a769c None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:58.723809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354302d-5312-45d8-80ab-208da41a769c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:58.723809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354302d-5312-45d8-80ab-208da41a769c None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:58.725456 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 678/1386] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:40:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:40:58.731100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99776e-bc07-47e8-9066-b3e9e6a62997 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:58.731634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99776e-bc07-47e8-9066-b3e9e6a62997 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:58.731634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99776e-bc07-47e8-9066-b3e9e6a62997 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:58.801878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-6d99776e-bc07-47e8-9066-b3e9e6a62997 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:40:58.803212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6d99776e-bc07-47e8-9066-b3e9e6a62997 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:58.839212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:58.903279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 709/1387] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:40:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:40:59.191483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354302d-5312-45d8-80ab-208da41a769c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:40:59.194424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c354302d-5312-45d8-80ab-208da41a769c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:59.285853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:59.370171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354302d-5312-45d8-80ab-208da41a769c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:40:59.491046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc48f26-4c12-4773-9eed-f0277b972f27 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:40:59.491779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc48f26-4c12-4773-9eed-f0277b972f27 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:40:59.492069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc48f26-4c12-4773-9eed-f0277b972f27 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:40:59.492352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc48f26-4c12-4773-9eed-f0277b972f27 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:40:59.494383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3dc48f26-4c12-4773-9eed-f0277b972f27 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:40:59.535872 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:40:59.591043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc48f26-4c12-4773-9eed-f0277b972f27 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:40:59.593226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc48f26-4c12-4773-9eed-f0277b972f27 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:40:59.618084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 679/1388] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:40:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:01.793918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad7c639-c8a2-40fe-8197-849278cd0321 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:01.794538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad7c639-c8a2-40fe-8197-849278cd0321 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:01.794538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad7c639-c8a2-40fe-8197-849278cd0321 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:01.795850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 710/1389] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:41:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:01.802437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c0aad-fd3e-4f74-b3e1-8290a911e4ed None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:01.803001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c0aad-fd3e-4f74-b3e1-8290a911e4ed None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:01.803101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c0aad-fd3e-4f74-b3e1-8290a911e4ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:01.871162 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-317c0aad-fd3e-4f74-b3e1-8290a911e4ed None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:01.872378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-317c0aad-fd3e-4f74-b3e1-8290a911e4ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:01.978038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 680/1390] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:41:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:02.041803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07b21bc-d189-44b4-994f-a8439087bca2 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:02.042090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07b21bc-d189-44b4-994f-a8439087bca2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:02.042399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07b21bc-d189-44b4-994f-a8439087bca2 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:02.043776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 711/1391] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:41:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:02.049155 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ca1bfe-ff03-4a50-a272-3ed344a92b4f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:02.049380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ca1bfe-ff03-4a50-a272-3ed344a92b4f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:02.049646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ca1bfe-ff03-4a50-a272-3ed344a92b4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:02.065186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:02.117791 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-14ca1bfe-ff03-4a50-a272-3ed344a92b4f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:02.119299 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-14ca1bfe-ff03-4a50-a272-3ed344a92b4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:02.224465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 681/1392] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:41:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:02.274205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07b21bc-d189-44b4-994f-a8439087bca2 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:02.275720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a07b21bc-d189-44b4-994f-a8439087bca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:02.424782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07b21bc-d189-44b4-994f-a8439087bca2 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:02.443560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:02.532228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491df04d-f618-4d22-af3d-9d63a2e40cc2 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:02.532567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491df04d-f618-4d22-af3d-9d63a2e40cc2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:02.532913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491df04d-f618-4d22-af3d-9d63a2e40cc2 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:02.534474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 682/1393] 10.222.0.65 () {54 vars in 967 bytes} [Wed Aug 24 03:41:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:02.540322 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cbf27e-8317-459d-ba4e-b9dda00842b6 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:02.542847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cbf27e-8317-459d-ba4e-b9dda00842b6 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:02.542847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cbf27e-8317-459d-ba4e-b9dda00842b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:02.560343 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3545c50c-c0aa-448c-804b-6cb60701309a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:02.560952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3545c50c-c0aa-448c-804b-6cb60701309a service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:02.561245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3545c50c-c0aa-448c-804b-6cb60701309a service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:02.561554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3545c50c-c0aa-448c-804b-6cb60701309a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:02.563168 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3545c50c-c0aa-448c-804b-6cb60701309a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:02.629309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-77cbf27e-8317-459d-ba4e-b9dda00842b6 None None] MFA Rules not processed for user `9f3b3c7ebc08496e970f8f25fd6395e4`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:02.631415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-77cbf27e-8317-459d-ba4e-b9dda00842b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:02.661961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3545c50c-c0aa-448c-804b-6cb60701309a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:02.663551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3545c50c-c0aa-448c-804b-6cb60701309a service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:02.674875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:02.683337 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 712/1394] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:02.723408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 683/1395] 10.222.0.65 () {58 vars in 1064 bytes} [Wed Aug 24 03:41:02 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:41:02.732151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3545c50c-c0aa-448c-804b-6cb60701309a service nova] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:02.733351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3545c50c-c0aa-448c-804b-6cb60701309a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:02.756428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:02.814462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3545c50c-c0aa-448c-804b-6cb60701309a service nova] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:02.891833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8ef696-219b-4c94-9b2f-18c798beeca9 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f3b3c7ebc08496e970f8f25fd6395e4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:02.892475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8ef696-219b-4c94-9b2f-18c798beeca9 service glance] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:02.892755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8ef696-219b-4c94-9b2f-18c798beeca9 service glance] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:02.893027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8ef696-219b-4c94-9b2f-18c798beeca9 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:02.894672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7f8ef696-219b-4c94-9b2f-18c798beeca9 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:03.006469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:03.020047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8ef696-219b-4c94-9b2f-18c798beeca9 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:03.022439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8ef696-219b-4c94-9b2f-18c798beeca9 service glance] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:03.043418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 713/1396] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:03.936642 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa14f4-fb99-403b-9407-e082ed3693e4 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:03.937092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa14f4-fb99-403b-9407-e082ed3693e4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:03.937239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa14f4-fb99-403b-9407-e082ed3693e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:04.010835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f3fa14f4-fb99-403b-9407-e082ed3693e4 None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:04.012042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f3fa14f4-fb99-403b-9407-e082ed3693e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:04.103407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 684/1397] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:41:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:04.149590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012177dd-96a6-4cd0-a9d8-3d69415e68b0 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:04.150124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012177dd-96a6-4cd0-a9d8-3d69415e68b0 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:04.150124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012177dd-96a6-4cd0-a9d8-3d69415e68b0 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:04.151522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 714/1398] 10.222.0.65 () {54 vars in 943 bytes} [Wed Aug 24 03:41:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:04.158220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d5d948-88ee-4f4d-8388-e50d377048c3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:04.158510 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d5d948-88ee-4f4d-8388-e50d377048c3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:04.158785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d5d948-88ee-4f4d-8388-e50d377048c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:04.219795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:04.239562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-76d5d948-88ee-4f4d-8388-e50d377048c3 None None] MFA Rules not processed for user `4591bec74a4641af818b2d75b657115e`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:04.240775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-76d5d948-88ee-4f4d-8388-e50d377048c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:04.350296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 685/1399] 10.222.0.65 () {58 vars in 1040 bytes} [Wed Aug 24 03:41:04 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:04.360614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012177dd-96a6-4cd0-a9d8-3d69415e68b0 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:04.362145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-012177dd-96a6-4cd0-a9d8-3d69415e68b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:04.507692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012177dd-96a6-4cd0-a9d8-3d69415e68b0 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:04.576667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:04.623545 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411e3786-b6ab-4936-bb29-e5f7dc501d3e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:04.624598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411e3786-b6ab-4936-bb29-e5f7dc501d3e service swift] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:04.625102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411e3786-b6ab-4936-bb29-e5f7dc501d3e service swift] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:04.625573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411e3786-b6ab-4936-bb29-e5f7dc501d3e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:04.627740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-411e3786-b6ab-4936-bb29-e5f7dc501d3e service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:04.715468 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411e3786-b6ab-4936-bb29-e5f7dc501d3e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:04.717060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411e3786-b6ab-4936-bb29-e5f7dc501d3e service swift] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:04.725490 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 715/1400] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:41:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:41:04.814526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d2b0e0-1e65-4138-9863-7767189c987c None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:04.814776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d2b0e0-1e65-4138-9863-7767189c987c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:04.815084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d2b0e0-1e65-4138-9863-7767189c987c None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:04.816714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 686/1401] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:41:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:04.823493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4f2f90-9685-46d5-9cff-e0d8666cf08e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:04.823782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4f2f90-9685-46d5-9cff-e0d8666cf08e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:04.824054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4f2f90-9685-46d5-9cff-e0d8666cf08e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:04.893782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-6c4f2f90-9685-46d5-9cff-e0d8666cf08e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:04.895016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6c4f2f90-9685-46d5-9cff-e0d8666cf08e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:04.948671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:04.996754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 716/1402] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:41:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:05.298002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d2b0e0-1e65-4138-9863-7767189c987c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:05.299910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a7d2b0e0-1e65-4138-9863-7767189c987c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:05.424516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:05.494622 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d2b0e0-1e65-4138-9863-7767189c987c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:05.644657 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d173ce62-2650-4b5e-afbd-cb9fb7cdf1ee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:05.645268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d173ce62-2650-4b5e-afbd-cb9fb7cdf1ee service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:05.646351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d173ce62-2650-4b5e-afbd-cb9fb7cdf1ee service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:05.646808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d173ce62-2650-4b5e-afbd-cb9fb7cdf1ee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:05.648741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d173ce62-2650-4b5e-afbd-cb9fb7cdf1ee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:05.751759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:05.777854 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d173ce62-2650-4b5e-afbd-cb9fb7cdf1ee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:05.779599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d173ce62-2650-4b5e-afbd-cb9fb7cdf1ee service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:05.802549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 687/1403] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 505 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:08.012620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a3205e-a1b4-488f-b494-2d086577e4a3 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:08.013206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a3205e-a1b4-488f-b494-2d086577e4a3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:08.013206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a3205e-a1b4-488f-b494-2d086577e4a3 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:08.014743 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 717/1404] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:41:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:08.019984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bac3dbb-1b06-4425-991a-045956f9d6fc None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:08.020496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bac3dbb-1b06-4425-991a-045956f9d6fc None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:08.020650 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bac3dbb-1b06-4425-991a-045956f9d6fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:08.092582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1bac3dbb-1b06-4425-991a-045956f9d6fc None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:08.093977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1bac3dbb-1b06-4425-991a-045956f9d6fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:08.221295 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 688/1405] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:41:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:08.520694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a3205e-a1b4-488f-b494-2d086577e4a3 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:08.522255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d7a3205e-a1b4-488f-b494-2d086577e4a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:08.700529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a3205e-a1b4-488f-b494-2d086577e4a3 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:08.744918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:08.836143 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc97134-a646-4183-acbc-0a07d2d12200 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:08.837123 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc97134-a646-4183-acbc-0a07d2d12200 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:08.837578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc97134-a646-4183-acbc-0a07d2d12200 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:08.838012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc97134-a646-4183-acbc-0a07d2d12200 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:08.839865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1dc97134-a646-4183-acbc-0a07d2d12200 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:08.949516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc97134-a646-4183-acbc-0a07d2d12200 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:08.951458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc97134-a646-4183-acbc-0a07d2d12200 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:08.971229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 718/1406] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:11.099821 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980194d0-2acc-4379-ba99-6afd8cc4902d None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:11.100397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980194d0-2acc-4379-ba99-6afd8cc4902d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:11.100397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980194d0-2acc-4379-ba99-6afd8cc4902d None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:11.102027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 689/1407] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:41:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:11.107218 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0dec37-e03d-45b7-b3e7-ec7d37a446ca None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:11.107643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0dec37-e03d-45b7-b3e7-ec7d37a446ca None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:11.107775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0dec37-e03d-45b7-b3e7-ec7d37a446ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:11.168485 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:11.184082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-3a0dec37-e03d-45b7-b3e7-ec7d37a446ca None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:11.185745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3a0dec37-e03d-45b7-b3e7-ec7d37a446ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:11.292782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 719/1408] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:41:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:11.596062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980194d0-2acc-4379-ba99-6afd8cc4902d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:11.597617 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-980194d0-2acc-4379-ba99-6afd8cc4902d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:11.661535 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:11.788357 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980194d0-2acc-4379-ba99-6afd8cc4902d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:11.924845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6386ad92-f56a-4d55-8dcc-3556d1aa5676 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:11.925538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6386ad92-f56a-4d55-8dcc-3556d1aa5676 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:11.925820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6386ad92-f56a-4d55-8dcc-3556d1aa5676 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:11.926091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6386ad92-f56a-4d55-8dcc-3556d1aa5676 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:11.927878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6386ad92-f56a-4d55-8dcc-3556d1aa5676 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:11.941398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:12.048164 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6386ad92-f56a-4d55-8dcc-3556d1aa5676 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:12.049710 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6386ad92-f56a-4d55-8dcc-3556d1aa5676 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:12.087088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 690/1409] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 491 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:21.290847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=46 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:22.081005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=41 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:29.552378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cd9964-2e6d-44f5-a403-f8b56108591a None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:29.552940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cd9964-2e6d-44f5-a403-f8b56108591a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:29.552940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cd9964-2e6d-44f5-a403-f8b56108591a None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:29.554269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 720/1410] 10.222.0.65 () {54 vars in 918 bytes} [Wed Aug 24 03:41:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:29.559672 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0679ff-3d9d-45d3-976b-760f5d37afad None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:29.560079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0679ff-3d9d-45d3-976b-760f5d37afad None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:29.560248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0679ff-3d9d-45d3-976b-760f5d37afad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:29.636845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-cf0679ff-3d9d-45d3-976b-760f5d37afad None None] MFA Rules not processed for user `4b347c9c89a64168b51c439b4a92d647`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:29.638061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-cf0679ff-3d9d-45d3-976b-760f5d37afad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:29.745424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 691/1411] 10.222.0.65 () {58 vars in 1015 bytes} [Wed Aug 24 03:41:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:29.759943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e147644c-8494-4b39-8f3d-fb01fdbb8462 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:29.760308 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e147644c-8494-4b39-8f3d-fb01fdbb8462 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:29.760641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e147644c-8494-4b39-8f3d-fb01fdbb8462 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:29.762064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 721/1412] 10.222.0.65 () {54 vars in 918 bytes} [Wed Aug 24 03:41:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:29.766692 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef9429e-fd13-49e5-a225-c57eef0ad933 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:29.767092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef9429e-fd13-49e5-a225-c57eef0ad933 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:29.767372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef9429e-fd13-49e5-a225-c57eef0ad933 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:29.841762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:29.844450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-5ef9429e-fd13-49e5-a225-c57eef0ad933 None None] MFA Rules not processed for user `e5896018712d4b82a781365e478ee855`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:29.845675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5ef9429e-fd13-49e5-a225-c57eef0ad933 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:29.962694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 692/1413] 10.222.0.65 () {58 vars in 1015 bytes} [Wed Aug 24 03:41:29 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:41:32.360724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174134b7-86d7-42ea-bb61-56c6b4573f91 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:32.361297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174134b7-86d7-42ea-bb61-56c6b4573f91 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:32.361297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174134b7-86d7-42ea-bb61-56c6b4573f91 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:32.362617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 722/1414] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:41:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:32.368018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d8eace-9335-4c9b-a3ec-795b99009c1f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:32.368687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d8eace-9335-4c9b-a3ec-795b99009c1f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:32.368852 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d8eace-9335-4c9b-a3ec-795b99009c1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:32.445759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-a5d8eace-9335-4c9b-a3ec-795b99009c1f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:32.447364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a5d8eace-9335-4c9b-a3ec-795b99009c1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:32.502919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:32.546837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 693/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:41:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:32.847744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174134b7-86d7-42ea-bb61-56c6b4573f91 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:32.849379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-174134b7-86d7-42ea-bb61-56c6b4573f91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:32.998263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174134b7-86d7-42ea-bb61-56c6b4573f91 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:33.073195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:33.116398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab461a3-0036-4ea2-adda-e32243e1e791 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:33.117006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab461a3-0036-4ea2-adda-e32243e1e791 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:33.117279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab461a3-0036-4ea2-adda-e32243e1e791 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:33.117543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab461a3-0036-4ea2-adda-e32243e1e791 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:33.119435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-fab461a3-0036-4ea2-adda-e32243e1e791 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:33.210636 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab461a3-0036-4ea2-adda-e32243e1e791 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:33.212236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab461a3-0036-4ea2-adda-e32243e1e791 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:33.231167 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 723/1416] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:34.129275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6f26fc-7fb1-42a6-8ae3-641b7aa8c348 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:34.129889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6f26fc-7fb1-42a6-8ae3-641b7aa8c348 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:34.130012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6f26fc-7fb1-42a6-8ae3-641b7aa8c348 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:34.131337 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 694/1417] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:41:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:34.136590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b37d22e-85af-440e-9c71-ed90865c731d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:34.137012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b37d22e-85af-440e-9c71-ed90865c731d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:34.137142 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b37d22e-85af-440e-9c71-ed90865c731d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:34.221971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1b37d22e-85af-440e-9c71-ed90865c731d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:34.221971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1b37d22e-85af-440e-9c71-ed90865c731d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:34.250746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:34.322715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 724/1418] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:41:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:34.482634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6f26fc-7fb1-42a6-8ae3-641b7aa8c348 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:34.485149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-bd6f26fc-7fb1-42a6-8ae3-641b7aa8c348 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:34.636104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6f26fc-7fb1-42a6-8ae3-641b7aa8c348 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:34.675912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:34.750931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3ae135-7957-4a2f-82ba-bb5db8151264 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:34.751458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3ae135-7957-4a2f-82ba-bb5db8151264 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:34.751736 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3ae135-7957-4a2f-82ba-bb5db8151264 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:34.752007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3ae135-7957-4a2f-82ba-bb5db8151264 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:34.753609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-af3ae135-7957-4a2f-82ba-bb5db8151264 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:34.843451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3ae135-7957-4a2f-82ba-bb5db8151264 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:34.845081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3ae135-7957-4a2f-82ba-bb5db8151264 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:34.863160 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 695/1419] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:35.762110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46039865-4e91-4616-971a-39e9ac78ca1e None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:35.762696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46039865-4e91-4616-971a-39e9ac78ca1e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:35.762696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46039865-4e91-4616-971a-39e9ac78ca1e None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:35.764091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 725/1420] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:41:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:35.769192 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c47721-e93f-4bef-a5c6-f2ddb922974f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:35.769634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c47721-e93f-4bef-a5c6-f2ddb922974f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:35.769843 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c47721-e93f-4bef-a5c6-f2ddb922974f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:35.828396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:35.850293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-e0c47721-e93f-4bef-a5c6-f2ddb922974f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:35.851516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-e0c47721-e93f-4bef-a5c6-f2ddb922974f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:35.948315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 696/1421] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:41:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:36.101591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46039865-4e91-4616-971a-39e9ac78ca1e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:36.103177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-46039865-4e91-4616-971a-39e9ac78ca1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:36.251200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46039865-4e91-4616-971a-39e9ac78ca1e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:36.263645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:36.364384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643c9102-9f4a-4a42-b846-7fec29b57837 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:36.364988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c9102-9f4a-4a42-b846-7fec29b57837 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:36.365263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c9102-9f4a-4a42-b846-7fec29b57837 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:36.365538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c9102-9f4a-4a42-b846-7fec29b57837 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:36.367172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-643c9102-9f4a-4a42-b846-7fec29b57837 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:36.458885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643c9102-9f4a-4a42-b846-7fec29b57837 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:36.460536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643c9102-9f4a-4a42-b846-7fec29b57837 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:36.479293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 726/1422] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:37.348019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02c2799-cbbb-4ecc-b6e6-f929ce59ed82 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:37.348881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02c2799-cbbb-4ecc-b6e6-f929ce59ed82 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:37.349068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02c2799-cbbb-4ecc-b6e6-f929ce59ed82 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:37.350625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 697/1423] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:41:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:37.355669 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95e7a96-f756-41ef-b562-2a5ef79c43f6 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:37.356099 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95e7a96-f756-41ef-b562-2a5ef79c43f6 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:37.356232 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95e7a96-f756-41ef-b562-2a5ef79c43f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:37.375829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:37.434670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c95e7a96-f756-41ef-b562-2a5ef79c43f6 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:37.436023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c95e7a96-f756-41ef-b562-2a5ef79c43f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:37.535702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 727/1424] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:41:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:37.690526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02c2799-cbbb-4ecc-b6e6-f929ce59ed82 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:37.692570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d02c2799-cbbb-4ecc-b6e6-f929ce59ed82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:37.800691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:37.836888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02c2799-cbbb-4ecc-b6e6-f929ce59ed82 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:37.950801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c1d4c8-4184-4829-bfcd-8015055217f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:37.951335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c1d4c8-4184-4829-bfcd-8015055217f4 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:37.951614 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c1d4c8-4184-4829-bfcd-8015055217f4 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:37.952297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c1d4c8-4184-4829-bfcd-8015055217f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:37.953471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-72c1d4c8-4184-4829-bfcd-8015055217f4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:38.029424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:38.043597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c1d4c8-4184-4829-bfcd-8015055217f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:38.045217 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c1d4c8-4184-4829-bfcd-8015055217f4 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:38.064175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 698/1425] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:38.935553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e0a9f3-6c82-4a0c-b9f5-c72a1052f2ac None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:38.936125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e0a9f3-6c82-4a0c-b9f5-c72a1052f2ac None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:38.936125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e0a9f3-6c82-4a0c-b9f5-c72a1052f2ac None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:38.937425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 728/1426] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:41:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:38.942980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae03de3b-8a6b-417a-8fe0-f07156591209 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:38.943405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae03de3b-8a6b-417a-8fe0-f07156591209 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:38.943520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae03de3b-8a6b-417a-8fe0-f07156591209 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:39.025384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-ae03de3b-8a6b-417a-8fe0-f07156591209 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:39.026634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ae03de3b-8a6b-417a-8fe0-f07156591209 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:39.125451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 699/1427] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:41:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:39.277845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e0a9f3-6c82-4a0c-b9f5-c72a1052f2ac None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:39.279311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b7e0a9f3-6c82-4a0c-b9f5-c72a1052f2ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:39.365139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:39.428549 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e0a9f3-6c82-4a0c-b9f5-c72a1052f2ac None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:39.543065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4867d0d8-faf4-422b-a024-ce1b83443542 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:39.543587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4867d0d8-faf4-422b-a024-ce1b83443542 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:39.543866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4867d0d8-faf4-422b-a024-ce1b83443542 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:39.544136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4867d0d8-faf4-422b-a024-ce1b83443542 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:39.545840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4867d0d8-faf4-422b-a024-ce1b83443542 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:39.594726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:39.639450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4867d0d8-faf4-422b-a024-ce1b83443542 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:39.641065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4867d0d8-faf4-422b-a024-ce1b83443542 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:39.659641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 729/1428] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:40.522469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd35418a-3cf4-4172-a5b3-6d21c34ae2ef None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:40.523026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd35418a-3cf4-4172-a5b3-6d21c34ae2ef None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:40.523026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd35418a-3cf4-4172-a5b3-6d21c34ae2ef None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:40.524481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 700/1429] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:41:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:40.529537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2970a7d-bed9-4645-8421-72f1906c3bca None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:40.529954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2970a7d-bed9-4645-8421-72f1906c3bca None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:40.530075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2970a7d-bed9-4645-8421-72f1906c3bca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:40.606930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-b2970a7d-bed9-4645-8421-72f1906c3bca None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:40.608181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b2970a7d-bed9-4645-8421-72f1906c3bca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:40.710071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 730/1430] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:41:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:40.860956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd35418a-3cf4-4172-a5b3-6d21c34ae2ef None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:40.862516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-bd35418a-3cf4-4172-a5b3-6d21c34ae2ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:40.914496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:41.011359 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd35418a-3cf4-4172-a5b3-6d21c34ae2ef None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:41.124536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485dd172-076f-4be0-8208-ff7cf3d7b4c0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:41.125136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485dd172-076f-4be0-8208-ff7cf3d7b4c0 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:41.125415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485dd172-076f-4be0-8208-ff7cf3d7b4c0 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:41.125686 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485dd172-076f-4be0-8208-ff7cf3d7b4c0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:41.127331 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-485dd172-076f-4be0-8208-ff7cf3d7b4c0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:41.135521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:41.219118 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485dd172-076f-4be0-8208-ff7cf3d7b4c0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:41.220726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485dd172-076f-4be0-8208-ff7cf3d7b4c0 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:41.238904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 701/1431] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:42.135330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb996d6-e3bd-46c9-b294-80e50b842116 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:42.135890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb996d6-e3bd-46c9-b294-80e50b842116 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:42.135890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb996d6-e3bd-46c9-b294-80e50b842116 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:42.137386 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 731/1432] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:41:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:42.142514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f36c31c-162a-42a2-8f21-07aa9d9b4dce None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:42.142939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f36c31c-162a-42a2-8f21-07aa9d9b4dce None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:42.143095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f36c31c-162a-42a2-8f21-07aa9d9b4dce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:42.227585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-3f36c31c-162a-42a2-8f21-07aa9d9b4dce None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:42.228712 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3f36c31c-162a-42a2-8f21-07aa9d9b4dce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:42.295895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:42.324123 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 702/1433] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:41:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:42.480159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb996d6-e3bd-46c9-b294-80e50b842116 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:42.481601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1fb996d6-e3bd-46c9-b294-80e50b842116 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:42.499666 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:42.634015 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb996d6-e3bd-46c9-b294-80e50b842116 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:42.721223 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:42.746637 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f068c5c-7826-47b0-b1c2-fb2ec064d8a7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:42.747178 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f068c5c-7826-47b0-b1c2-fb2ec064d8a7 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:42.747452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f068c5c-7826-47b0-b1c2-fb2ec064d8a7 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:42.747727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f068c5c-7826-47b0-b1c2-fb2ec064d8a7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:42.750009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8f068c5c-7826-47b0-b1c2-fb2ec064d8a7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:42.841603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f068c5c-7826-47b0-b1c2-fb2ec064d8a7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:42.843260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f068c5c-7826-47b0-b1c2-fb2ec064d8a7 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:42.861099 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 732/1434] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:43.735133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e8f898-827f-408c-91c0-323ec383cc57 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:43.735711 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e8f898-827f-408c-91c0-323ec383cc57 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:43.735711 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e8f898-827f-408c-91c0-323ec383cc57 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:43.736940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 703/1435] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:41:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:43.742522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14deb2f7-4d2f-445b-b560-374a9f05c399 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:43.742979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14deb2f7-4d2f-445b-b560-374a9f05c399 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:43.743368 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14deb2f7-4d2f-445b-b560-374a9f05c399 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:43.824206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-14deb2f7-4d2f-445b-b560-374a9f05c399 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:43.825471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-14deb2f7-4d2f-445b-b560-374a9f05c399 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:43.868667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:43.923753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 733/1436] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:41:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:44.076920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e8f898-827f-408c-91c0-323ec383cc57 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:44.078460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-18e8f898-827f-408c-91c0-323ec383cc57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:44.224557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e8f898-827f-408c-91c0-323ec383cc57 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:44.281886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:44.342444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcd0ffd-9a12-4963-a68b-36aa81ba51c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:44.343170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcd0ffd-9a12-4963-a68b-36aa81ba51c6 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:44.343710 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcd0ffd-9a12-4963-a68b-36aa81ba51c6 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:44.344394 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcd0ffd-9a12-4963-a68b-36aa81ba51c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:44.346111 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ddcd0ffd-9a12-4963-a68b-36aa81ba51c6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:44.442961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcd0ffd-9a12-4963-a68b-36aa81ba51c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:44.444568 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcd0ffd-9a12-4963-a68b-36aa81ba51c6 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:44.463666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 704/1437] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:45.370737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a75a683-a629-4cd9-b500-79f0e7d8f78d None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:45.371330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a75a683-a629-4cd9-b500-79f0e7d8f78d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:45.371330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a75a683-a629-4cd9-b500-79f0e7d8f78d None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:45.372627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 734/1438] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:41:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:45.377716 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4dfc5e-fe75-43a4-9de9-c8968d0be044 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:45.378174 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4dfc5e-fe75-43a4-9de9-c8968d0be044 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:45.378302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4dfc5e-fe75-43a4-9de9-c8968d0be044 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:45.459410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-2a4dfc5e-fe75-43a4-9de9-c8968d0be044 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:45.460653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-2a4dfc5e-fe75-43a4-9de9-c8968d0be044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:45.462709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:45.561626 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 705/1439] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:41:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:45.713699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a75a683-a629-4cd9-b500-79f0e7d8f78d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:45.715237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3a75a683-a629-4cd9-b500-79f0e7d8f78d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:45.861852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a75a683-a629-4cd9-b500-79f0e7d8f78d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:45.890135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:45.978888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b129532a-4128-4ff6-8ed1-0bfcd43b7a02 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:45.979399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b129532a-4128-4ff6-8ed1-0bfcd43b7a02 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:45.979863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b129532a-4128-4ff6-8ed1-0bfcd43b7a02 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:45.980152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b129532a-4128-4ff6-8ed1-0bfcd43b7a02 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:45.981762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b129532a-4128-4ff6-8ed1-0bfcd43b7a02 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:46.071996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b129532a-4128-4ff6-8ed1-0bfcd43b7a02 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:46.073598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b129532a-4128-4ff6-8ed1-0bfcd43b7a02 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:46.092188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 735/1440] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:47.966109 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1752c36-d92e-4367-803f-e18a5d87b51c None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:47.966585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1752c36-d92e-4367-803f-e18a5d87b51c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:47.966686 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1752c36-d92e-4367-803f-e18a5d87b51c None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:47.968035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 706/1441] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:41:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:47.974328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3b34f4-21da-4532-848a-64f505afab2a None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:47.974823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3b34f4-21da-4532-848a-64f505afab2a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:47.974961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3b34f4-21da-4532-848a-64f505afab2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:48.017246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:48.042786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-ae3b34f4-21da-4532-848a-64f505afab2a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:48.044062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ae3b34f4-21da-4532-848a-64f505afab2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:48.141261 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 736/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:41:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:48.424417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1752c36-d92e-4367-803f-e18a5d87b51c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:48.426688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f1752c36-d92e-4367-803f-e18a5d87b51c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:48.536673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1752c36-d92e-4367-803f-e18a5d87b51c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:48.591631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:48.626102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b874190-3690-489a-857c-47a30d528dd4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:48.626779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b874190-3690-489a-857c-47a30d528dd4 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:48.627135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b874190-3690-489a-857c-47a30d528dd4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:48.627403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b874190-3690-489a-857c-47a30d528dd4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:48.629389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4b874190-3690-489a-857c-47a30d528dd4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:48.747037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b874190-3690-489a-857c-47a30d528dd4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:48.748522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b874190-3690-489a-857c-47a30d528dd4 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:48.767232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 707/1443] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:50.895193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e3b8e8-b6f4-432d-8725-e5d29cb63110 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:50.896016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e3b8e8-b6f4-432d-8725-e5d29cb63110 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:50.896016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e3b8e8-b6f4-432d-8725-e5d29cb63110 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:50.897321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 737/1444] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:41:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:50.903066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919dd2d6-7a25-47f0-94d7-eff90ed5e6ae None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:50.903775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919dd2d6-7a25-47f0-94d7-eff90ed5e6ae None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:50.904040 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919dd2d6-7a25-47f0-94d7-eff90ed5e6ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:50.976539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-919dd2d6-7a25-47f0-94d7-eff90ed5e6ae None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:50.977756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-919dd2d6-7a25-47f0-94d7-eff90ed5e6ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:50.991198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:51.079065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 708/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:41:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:41:51.368402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e3b8e8-b6f4-432d-8725-e5d29cb63110 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:51.369952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e2e3b8e8-b6f4-432d-8725-e5d29cb63110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:51.481911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e3b8e8-b6f4-432d-8725-e5d29cb63110 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:51.495792 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:51.560465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10724f68-392f-4548-9398-6df6cda127b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:51.561074 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10724f68-392f-4548-9398-6df6cda127b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:51.561348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10724f68-392f-4548-9398-6df6cda127b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:51.561615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10724f68-392f-4548-9398-6df6cda127b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:51.563462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-10724f68-392f-4548-9398-6df6cda127b0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:51.678839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10724f68-392f-4548-9398-6df6cda127b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:51.680193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10724f68-392f-4548-9398-6df6cda127b0 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:51.698858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 738/1446] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:54.056410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a11f22-b0de-4ba5-a9de-e74127e4a469 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:54.056967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a11f22-b0de-4ba5-a9de-e74127e4a469 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:54.056967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a11f22-b0de-4ba5-a9de-e74127e4a469 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:54.058341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 709/1447] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:41:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:54.063934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f0e4c-5281-4f6a-836a-5125288fb37f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:54.064399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f0e4c-5281-4f6a-836a-5125288fb37f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:54.064497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f0e4c-5281-4f6a-836a-5125288fb37f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:54.113768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:54.132171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-6a6f0e4c-5281-4f6a-836a-5125288fb37f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:54.133401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6a6f0e4c-5281-4f6a-836a-5125288fb37f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:54.234595 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 739/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:41:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:41:54.523814 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a11f22-b0de-4ba5-a9de-e74127e4a469 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:54.525434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-83a11f22-b0de-4ba5-a9de-e74127e4a469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:54.632970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a11f22-b0de-4ba5-a9de-e74127e4a469 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:54.676206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:54.725045 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b1e590-5936-4c70-b4c5-63fc701f87ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:54.725946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b1e590-5936-4c70-b4c5-63fc701f87ad service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:54.726416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b1e590-5936-4c70-b4c5-63fc701f87ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:54.726836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b1e590-5936-4c70-b4c5-63fc701f87ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:54.729440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-95b1e590-5936-4c70-b4c5-63fc701f87ad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:54.850380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b1e590-5936-4c70-b4c5-63fc701f87ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:54.851861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b1e590-5936-4c70-b4c5-63fc701f87ad service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:54.871497 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 710/1449] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:56.996984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d1f38c-e260-44b4-b57d-3a08bdb60a07 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:56.997573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d1f38c-e260-44b4-b57d-3a08bdb60a07 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:56.997573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d1f38c-e260-44b4-b57d-3a08bdb60a07 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:56.998925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 740/1450] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:41:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:57.005024 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1d0517-2fbb-452a-9015-c642145b38f9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:57.007845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1d0517-2fbb-452a-9015-c642145b38f9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:57.007845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1d0517-2fbb-452a-9015-c642145b38f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:57.074942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-db1d0517-2fbb-452a-9015-c642145b38f9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:57.076155 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-db1d0517-2fbb-452a-9015-c642145b38f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:57.097068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:57.295800 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 711/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:41:57.587598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d1f38c-e260-44b4-b57d-3a08bdb60a07 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:41:57.589226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-68d1f38c-e260-44b4-b57d-3a08bdb60a07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:57.694273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d1f38c-e260-44b4-b57d-3a08bdb60a07 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:41:57.713783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:57.768808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f472807-2540-4302-bc64-a28b5ba2838e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:41:57.769508 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f472807-2540-4302-bc64-a28b5ba2838e service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:57.769800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f472807-2540-4302-bc64-a28b5ba2838e service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:57.770070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f472807-2540-4302-bc64-a28b5ba2838e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:57.771706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1f472807-2540-4302-bc64-a28b5ba2838e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:41:57.882532 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f472807-2540-4302-bc64-a28b5ba2838e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:41:57.883977 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f472807-2540-4302-bc64-a28b5ba2838e service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:41:57.903375 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 741/1452] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:41:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:41:59.904393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d20d85-9e5f-45cd-a23a-21ee2b8328af None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:59.904993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d20d85-9e5f-45cd-a23a-21ee2b8328af None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:59.904993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d20d85-9e5f-45cd-a23a-21ee2b8328af None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:59.906458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 712/1453] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:41:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:41:59.912419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd83b51-6f42-4cf0-8530-c10951948326 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:41:59.912844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd83b51-6f42-4cf0-8530-c10951948326 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:41:59.913072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd83b51-6f42-4cf0-8530-c10951948326 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:41:59.980961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:41:59.985117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-8bd83b51-6f42-4cf0-8530-c10951948326 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:41:59.986493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8bd83b51-6f42-4cf0-8530-c10951948326 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:00.090686 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 742/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:41:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:00.380108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d20d85-9e5f-45cd-a23a-21ee2b8328af None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:00.381820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-43d20d85-9e5f-45cd-a23a-21ee2b8328af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:00.494030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d20d85-9e5f-45cd-a23a-21ee2b8328af None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:00.544271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:00.569583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a12b477-455a-4933-8896-ceb394015308 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:00.570210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a12b477-455a-4933-8896-ceb394015308 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:00.570488 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a12b477-455a-4933-8896-ceb394015308 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:00.570756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a12b477-455a-4933-8896-ceb394015308 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:00.572356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5a12b477-455a-4933-8896-ceb394015308 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:00.681803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a12b477-455a-4933-8896-ceb394015308 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:00.683183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a12b477-455a-4933-8896-ceb394015308 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:00.701827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 713/1455] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:42:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:42:02.712490 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc1ee2a-30c4-468e-94eb-aada266bb4f9 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:02.713045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc1ee2a-30c4-468e-94eb-aada266bb4f9 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:02.713045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc1ee2a-30c4-468e-94eb-aada266bb4f9 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:02.714512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 743/1456] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:42:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:42:02.719955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026f8ea5-02a8-4cac-8813-d09180783f8f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:02.720405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026f8ea5-02a8-4cac-8813-d09180783f8f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:02.720528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026f8ea5-02a8-4cac-8813-d09180783f8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:02.789479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-026f8ea5-02a8-4cac-8813-d09180783f8f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:02.790754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-026f8ea5-02a8-4cac-8813-d09180783f8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:02.822009 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:02.893674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 714/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:42:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:03.183093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc1ee2a-30c4-468e-94eb-aada266bb4f9 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:03.185148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1fc1ee2a-30c4-468e-94eb-aada266bb4f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:03.295443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc1ee2a-30c4-468e-94eb-aada266bb4f9 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:03.325690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:03.373135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26518941-41d7-49ff-a8c8-832bb73f210a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:03.373738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26518941-41d7-49ff-a8c8-832bb73f210a service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:03.374018 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26518941-41d7-49ff-a8c8-832bb73f210a service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:03.374338 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26518941-41d7-49ff-a8c8-832bb73f210a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:03.375967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-26518941-41d7-49ff-a8c8-832bb73f210a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:03.490757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26518941-41d7-49ff-a8c8-832bb73f210a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:03.492201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26518941-41d7-49ff-a8c8-832bb73f210a service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:03.512402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 744/1458] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:42:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:42:05.526395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e991933-f109-4192-9398-5fb3219475c2 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:05.526966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e991933-f109-4192-9398-5fb3219475c2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:05.526966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e991933-f109-4192-9398-5fb3219475c2 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:05.528391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 715/1459] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:42:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:42:05.534186 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c23a36-527a-4027-a101-f24a12748a50 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:05.535536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c23a36-527a-4027-a101-f24a12748a50 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:05.536040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c23a36-527a-4027-a101-f24a12748a50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:05.605364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-53c23a36-527a-4027-a101-f24a12748a50 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:05.606664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-53c23a36-527a-4027-a101-f24a12748a50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:05.616710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:05.715185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 745/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:42:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:06.002211 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e991933-f109-4192-9398-5fb3219475c2 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:06.004900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-2e991933-f109-4192-9398-5fb3219475c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:06.110630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e991933-f109-4192-9398-5fb3219475c2 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:06.171450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:06.190463 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029938b1-a3f9-4fd5-ab18-8f4f8f93622b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:06.192355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029938b1-a3f9-4fd5-ab18-8f4f8f93622b service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:06.192355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029938b1-a3f9-4fd5-ab18-8f4f8f93622b service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:06.192355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029938b1-a3f9-4fd5-ab18-8f4f8f93622b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:06.193847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-029938b1-a3f9-4fd5-ab18-8f4f8f93622b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:06.306311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029938b1-a3f9-4fd5-ab18-8f4f8f93622b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:06.307669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029938b1-a3f9-4fd5-ab18-8f4f8f93622b service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:06.326348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 716/1461] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:42:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:42:08.380180 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05133e5c-761c-430b-b02b-ee25bd376aa9 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:08.380791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05133e5c-761c-430b-b02b-ee25bd376aa9 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:08.380791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05133e5c-761c-430b-b02b-ee25bd376aa9 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:08.382073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 746/1462] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:42:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:42:08.387745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f462cac9-083c-4c09-9e25-76c96073d422 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:08.388261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f462cac9-083c-4c09-9e25-76c96073d422 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:08.388261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f462cac9-083c-4c09-9e25-76c96073d422 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:08.458378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f462cac9-083c-4c09-9e25-76c96073d422 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:08.459588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f462cac9-083c-4c09-9e25-76c96073d422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:08.499670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:08.560691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 717/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:42:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:42:08.845971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05133e5c-761c-430b-b02b-ee25bd376aa9 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:08.848129 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-05133e5c-761c-430b-b02b-ee25bd376aa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:08.956511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05133e5c-761c-430b-b02b-ee25bd376aa9 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:08.994541 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:09.032064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2326c9b4-9cf8-4b55-ac81-adc2193c9d1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:09.032668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2326c9b4-9cf8-4b55-ac81-adc2193c9d1e service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:09.032971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2326c9b4-9cf8-4b55-ac81-adc2193c9d1e service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:09.033240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2326c9b4-9cf8-4b55-ac81-adc2193c9d1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:09.035505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-2326c9b4-9cf8-4b55-ac81-adc2193c9d1e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:09.145704 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2326c9b4-9cf8-4b55-ac81-adc2193c9d1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:09.147177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2326c9b4-9cf8-4b55-ac81-adc2193c9d1e service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:09.169420 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 747/1464] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:42:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:42:11.155640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54389c65-080b-4c06-a189-f929a4230a0b None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:11.156218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54389c65-080b-4c06-a189-f929a4230a0b None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:11.156218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54389c65-080b-4c06-a189-f929a4230a0b None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:11.157568 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 718/1465] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:42:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:42:11.163263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2094179-ec6d-4e0d-939e-41a6f282b336 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:11.163719 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2094179-ec6d-4e0d-939e-41a6f282b336 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:11.163827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2094179-ec6d-4e0d-939e-41a6f282b336 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:11.231320 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c2094179-ec6d-4e0d-939e-41a6f282b336 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:11.232568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c2094179-ec6d-4e0d-939e-41a6f282b336 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:11.251800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:11.333194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 748/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:42:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:11.619041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54389c65-080b-4c06-a189-f929a4230a0b None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:11.620701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-54389c65-080b-4c06-a189-f929a4230a0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:11.725529 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54389c65-080b-4c06-a189-f929a4230a0b None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:11.792795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:11.799820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1109b164-c87f-4de5-b2f4-c3819eac84d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:11.800681 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1109b164-c87f-4de5-b2f4-c3819eac84d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:11.801329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1109b164-c87f-4de5-b2f4-c3819eac84d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:11.801329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1109b164-c87f-4de5-b2f4-c3819eac84d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:11.802821 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1109b164-c87f-4de5-b2f4-c3819eac84d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:11.940666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1109b164-c87f-4de5-b2f4-c3819eac84d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:11.942127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1109b164-c87f-4de5-b2f4-c3819eac84d6 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:11.964143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 719/1467] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:42:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:42:13.930239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be549ddf-bb90-4fab-ad22-f4679097cd2e None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:13.930892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be549ddf-bb90-4fab-ad22-f4679097cd2e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:13.930892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be549ddf-bb90-4fab-ad22-f4679097cd2e None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:13.932173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 749/1468] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:42:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:42:13.937855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede67f8f-abdf-4511-995e-cdb60fab2596 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:13.938380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede67f8f-abdf-4511-995e-cdb60fab2596 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:13.938508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede67f8f-abdf-4511-995e-cdb60fab2596 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:14.013265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-ede67f8f-abdf-4511-995e-cdb60fab2596 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:14.015083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ede67f8f-abdf-4511-995e-cdb60fab2596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:14.090386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:14.162774 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 720/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:42:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:14.456750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be549ddf-bb90-4fab-ad22-f4679097cd2e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:14.458727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-be549ddf-bb90-4fab-ad22-f4679097cd2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:14.572111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be549ddf-bb90-4fab-ad22-f4679097cd2e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:14.626871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:14.651082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84bf579-9ae6-49f5-80eb-391be345278f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:14.651690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84bf579-9ae6-49f5-80eb-391be345278f service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:14.651970 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84bf579-9ae6-49f5-80eb-391be345278f service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:14.652239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84bf579-9ae6-49f5-80eb-391be345278f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:14.653866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b84bf579-9ae6-49f5-80eb-391be345278f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:14.763016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84bf579-9ae6-49f5-80eb-391be345278f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:14.764377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84bf579-9ae6-49f5-80eb-391be345278f service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:14.783384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 750/1470] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:42:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:42:24.160876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=22 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:24.781118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=66 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:34.843376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a8ed4d-e287-463a-84dd-f510de038fc4 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:34.843905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a8ed4d-e287-463a-84dd-f510de038fc4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:34.843905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a8ed4d-e287-463a-84dd-f510de038fc4 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:34.845285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 721/1471] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:42:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:42:34.850657 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26684a3e-dfe5-4003-b969-60a9424f37d8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:34.851108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26684a3e-dfe5-4003-b969-60a9424f37d8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:34.851206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26684a3e-dfe5-4003-b969-60a9424f37d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:34.916110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-26684a3e-dfe5-4003-b969-60a9424f37d8 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:34.917328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-26684a3e-dfe5-4003-b969-60a9424f37d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:35.017823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 751/1472] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:42:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:35.314587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492beeea-3459-44b9-bb87-f28c35719f5a None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:35.314893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492beeea-3459-44b9-bb87-f28c35719f5a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:35.315163 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492beeea-3459-44b9-bb87-f28c35719f5a None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:35.317370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 722/1473] 10.222.0.65 () {54 vars in 967 bytes} [Wed Aug 24 03:42:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:42:35.322402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968e08cf-30d5-4f09-bc59-09b9beec2fa8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:35.322721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968e08cf-30d5-4f09-bc59-09b9beec2fa8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:35.322999 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968e08cf-30d5-4f09-bc59-09b9beec2fa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:35.404036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-968e08cf-30d5-4f09-bc59-09b9beec2fa8 None None] MFA Rules not processed for user `9f3b3c7ebc08496e970f8f25fd6395e4`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:35.405294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-968e08cf-30d5-4f09-bc59-09b9beec2fa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:35.426580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:35.482871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 752/1474] 10.222.0.65 () {58 vars in 1064 bytes} [Wed Aug 24 03:42:35 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:35.490500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492beeea-3459-44b9-bb87-f28c35719f5a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:35.492050 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-492beeea-3459-44b9-bb87-f28c35719f5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:35.605274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492beeea-3459-44b9-bb87-f28c35719f5a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:35.680924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49b4fb1-286b-4c9c-91c0-fdbc9cac18d0 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f3b3c7ebc08496e970f8f25fd6395e4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:35.681525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49b4fb1-286b-4c9c-91c0-fdbc9cac18d0 service glance] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:35.681798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49b4fb1-286b-4c9c-91c0-fdbc9cac18d0 service glance] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:35.682082 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49b4fb1-286b-4c9c-91c0-fdbc9cac18d0 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:35.683682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f49b4fb1-286b-4c9c-91c0-fdbc9cac18d0 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:35.738889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:35.798080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49b4fb1-286b-4c9c-91c0-fdbc9cac18d0 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:35.799546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49b4fb1-286b-4c9c-91c0-fdbc9cac18d0 service glance] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:35.819439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 723/1475] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:42:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:42:37.810901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858b8f60-31cc-473f-b438-959858dfa0cc None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:37.811411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858b8f60-31cc-473f-b438-959858dfa0cc None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:37.811411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858b8f60-31cc-473f-b438-959858dfa0cc None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:37.812696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 753/1476] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:42:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:42:37.818266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de6f40a-e105-472d-b071-061f4eafb893 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:37.818694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de6f40a-e105-472d-b071-061f4eafb893 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:37.818849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de6f40a-e105-472d-b071-061f4eafb893 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:37.884900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-3de6f40a-e105-472d-b071-061f4eafb893 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:37.886128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3de6f40a-e105-472d-b071-061f4eafb893 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:37.981902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 724/1477] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:42:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:38.257229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858b8f60-31cc-473f-b438-959858dfa0cc None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:38.258864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-858b8f60-31cc-473f-b438-959858dfa0cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:38.395885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858b8f60-31cc-473f-b438-959858dfa0cc None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:38.447096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:38.492332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f817d5-e532-4cd5-9800-d180ffe824fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:38.493197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f817d5-e532-4cd5-9800-d180ffe824fc admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:38.493480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f817d5-e532-4cd5-9800-d180ffe824fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:38.493781 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f817d5-e532-4cd5-9800-d180ffe824fc admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:38.494450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f817d5-e532-4cd5-9800-d180ffe824fc admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:38.496229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: WARNING py.warnings [None req-b0f817d5-e532-4cd5-9800-d180ffe824fc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:42:38.496229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: warnings.warn(msg) Aug 24 03:42:38.496229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: Aug 24 03:42:38.497378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f817d5-e532-4cd5-9800-d180ffe824fc admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:38.517072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 754/1478] 10.222.0.65 () {56 vars in 1108 bytes} [Wed Aug 24 03:42:38 2022] GET /identity/v3/projects => generated 2553 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:42:40.480359 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d97b9-c3a7-4324-954a-70aed7186ae9 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:40.480890 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d97b9-c3a7-4324-954a-70aed7186ae9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:40.480890 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d97b9-c3a7-4324-954a-70aed7186ae9 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:40.482269 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 725/1479] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:42:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:42:40.487560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec568216-79af-438f-9837-0695064c1b78 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:40.487988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec568216-79af-438f-9837-0695064c1b78 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:40.488106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec568216-79af-438f-9837-0695064c1b78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:40.553861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-ec568216-79af-438f-9837-0695064c1b78 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:40.555115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ec568216-79af-438f-9837-0695064c1b78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:40.653764 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 755/1480] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:42:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:40.947227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897d97b9-c3a7-4324-954a-70aed7186ae9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:40.948782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-897d97b9-c3a7-4324-954a-70aed7186ae9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:40.958565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:41.116279 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897d97b9-c3a7-4324-954a-70aed7186ae9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:41.197501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:41.231429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce13dd45-86b5-4177-bd2b-dd0adb170115 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:41.232043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce13dd45-86b5-4177-bd2b-dd0adb170115 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:41.232322 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce13dd45-86b5-4177-bd2b-dd0adb170115 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:41.232611 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce13dd45-86b5-4177-bd2b-dd0adb170115 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:41.235061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ce13dd45-86b5-4177-bd2b-dd0adb170115 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:41.332750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce13dd45-86b5-4177-bd2b-dd0adb170115 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:41.334384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce13dd45-86b5-4177-bd2b-dd0adb170115 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:41.353533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 726/1481] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:42:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:42:43.498033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7adbdda-dba1-4bdc-a6ce-ee4058e00927 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:43.498769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7adbdda-dba1-4bdc-a6ce-ee4058e00927 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:43.498769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7adbdda-dba1-4bdc-a6ce-ee4058e00927 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:43.499951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 756/1482] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:42:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:42:43.506687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e258f57-f3aa-4e6c-8a80-d2b62e9cbe39 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:43.508396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e258f57-f3aa-4e6c-8a80-d2b62e9cbe39 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:43.509352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e258f57-f3aa-4e6c-8a80-d2b62e9cbe39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:43.577636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-2e258f57-f3aa-4e6c-8a80-d2b62e9cbe39 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:43.578898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-2e258f57-f3aa-4e6c-8a80-d2b62e9cbe39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:43.626458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:43.683248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 727/1483] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:42:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:43.976776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7adbdda-dba1-4bdc-a6ce-ee4058e00927 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:43.978448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b7adbdda-dba1-4bdc-a6ce-ee4058e00927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:44.005533 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:44.085689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7adbdda-dba1-4bdc-a6ce-ee4058e00927 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:44.159701 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8b645a-0f1b-4cf0-86d8-a7868517af71 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:44.160319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8b645a-0f1b-4cf0-86d8-a7868517af71 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:44.160599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8b645a-0f1b-4cf0-86d8-a7868517af71 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:44.160869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8b645a-0f1b-4cf0-86d8-a7868517af71 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:44.162590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9d8b645a-0f1b-4cf0-86d8-a7868517af71 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:44.239493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:44.276695 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8b645a-0f1b-4cf0-86d8-a7868517af71 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:44.278177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8b645a-0f1b-4cf0-86d8-a7868517af71 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:44.297905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 757/1484] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:42:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:42:46.275205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a761adc-5fd0-49e1-a496-51ca5b6981ac None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:46.275745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a761adc-5fd0-49e1-a496-51ca5b6981ac None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:46.275745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a761adc-5fd0-49e1-a496-51ca5b6981ac None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:46.277017 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 728/1485] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:42:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:42:46.282502 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3e88b9-fbff-4a57-9b8b-5dde74e6f17a None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:46.282975 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3e88b9-fbff-4a57-9b8b-5dde74e6f17a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:46.283100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3e88b9-fbff-4a57-9b8b-5dde74e6f17a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:46.352053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-3c3e88b9-fbff-4a57-9b8b-5dde74e6f17a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:46.353412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3c3e88b9-fbff-4a57-9b8b-5dde74e6f17a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:46.456246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 758/1486] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:42:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:46.743729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a761adc-5fd0-49e1-a496-51ca5b6981ac None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:46.745291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6a761adc-5fd0-49e1-a496-51ca5b6981ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:46.851901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a761adc-5fd0-49e1-a496-51ca5b6981ac None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:46.925899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:46.927773 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa05418-0514-4fe4-b6fa-c7bd2c1b1abc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:46.928381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa05418-0514-4fe4-b6fa-c7bd2c1b1abc service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:46.928655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa05418-0514-4fe4-b6fa-c7bd2c1b1abc service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:46.928924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa05418-0514-4fe4-b6fa-c7bd2c1b1abc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:46.930643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7aa05418-0514-4fe4-b6fa-c7bd2c1b1abc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:47.046612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa05418-0514-4fe4-b6fa-c7bd2c1b1abc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:47.048128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa05418-0514-4fe4-b6fa-c7bd2c1b1abc service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:47.069331 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 729/1487] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:42:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:42:49.188569 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf7b299-0c0d-41cc-8e90-bb07b4b63da1 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:49.189106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf7b299-0c0d-41cc-8e90-bb07b4b63da1 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:49.189106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf7b299-0c0d-41cc-8e90-bb07b4b63da1 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:49.190549 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 759/1488] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:42:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:42:49.196280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e538ce4-2289-4ea9-a68a-7f0cd12af474 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:49.196885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e538ce4-2289-4ea9-a68a-7f0cd12af474 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:49.196885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e538ce4-2289-4ea9-a68a-7f0cd12af474 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:49.263981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-5e538ce4-2289-4ea9-a68a-7f0cd12af474 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:49.265267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5e538ce4-2289-4ea9-a68a-7f0cd12af474 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:49.319052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:49.364271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 730/1489] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:42:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:49.655181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf7b299-0c0d-41cc-8e90-bb07b4b63da1 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:49.656736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-eaf7b299-0c0d-41cc-8e90-bb07b4b63da1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:49.699941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:49.771878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf7b299-0c0d-41cc-8e90-bb07b4b63da1 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:49.848362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2eaae7-79ce-46b5-aa72-d7e9bf6b0bd7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:49.848963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2eaae7-79ce-46b5-aa72-d7e9bf6b0bd7 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:49.849242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2eaae7-79ce-46b5-aa72-d7e9bf6b0bd7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:49.849514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2eaae7-79ce-46b5-aa72-d7e9bf6b0bd7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:49.851284 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6b2eaae7-79ce-46b5-aa72-d7e9bf6b0bd7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:49.929794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:49.959993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2eaae7-79ce-46b5-aa72-d7e9bf6b0bd7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:49.961425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2eaae7-79ce-46b5-aa72-d7e9bf6b0bd7 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:49.980197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 760/1490] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:42:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:42:52.554701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9625d5a9-f88a-4db3-b454-277d64ace9d6 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:52.555302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9625d5a9-f88a-4db3-b454-277d64ace9d6 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:52.555302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9625d5a9-f88a-4db3-b454-277d64ace9d6 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:52.556570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 731/1491] 10.222.0.65 () {54 vars in 927 bytes} [Wed Aug 24 03:42:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:42:52.561936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54169d38-fdd4-4073-a521-7e69ea690a60 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:52.562437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54169d38-fdd4-4073-a521-7e69ea690a60 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:52.562663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54169d38-fdd4-4073-a521-7e69ea690a60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:52.634700 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-54169d38-fdd4-4073-a521-7e69ea690a60 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:42:52.635933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-54169d38-fdd4-4073-a521-7e69ea690a60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:52.732976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 761/1492] 10.222.0.65 () {58 vars in 1024 bytes} [Wed Aug 24 03:42:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:42:53.023658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9625d5a9-f88a-4db3-b454-277d64ace9d6 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:42:53.025189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-9625d5a9-f88a-4db3-b454-277d64ace9d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:53.129376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9625d5a9-f88a-4db3-b454-277d64ace9d6 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:42:53.202587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5743312d-58ec-463e-89f8-5c4a42a05cbc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:42:53.203187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743312d-58ec-463e-89f8-5c4a42a05cbc service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:42:53.203462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743312d-58ec-463e-89f8-5c4a42a05cbc service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:42:53.203730 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743312d-58ec-463e-89f8-5c4a42a05cbc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:42:53.205316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5743312d-58ec-463e-89f8-5c4a42a05cbc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:42:53.214900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:42:53.316670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5743312d-58ec-463e-89f8-5c4a42a05cbc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:42:53.318062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5743312d-58ec-463e-89f8-5c4a42a05cbc service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:42:53.336641 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 732/1493] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:42:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:43:02.731224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=82 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:43:03.334512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=50 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:44:13.560296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff649bf-3cbc-4b34-bfe4-72e28bccdbeb None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:13.560889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff649bf-3cbc-4b34-bfe4-72e28bccdbeb None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:13.560889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff649bf-3cbc-4b34-bfe4-72e28bccdbeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:13.635930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1ff649bf-3cbc-4b34-bfe4-72e28bccdbeb None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:44:13.637154 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1ff649bf-3cbc-4b34-bfe4-72e28bccdbeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:13.734249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 762/1494] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:44:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:44:13.743583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5389d6-cbf6-4a25-99b7-46944e1569d3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:13.744096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5389d6-cbf6-4a25-99b7-46944e1569d3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:13.744199 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5389d6-cbf6-4a25-99b7-46944e1569d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:13.820114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1e5389d6-cbf6-4a25-99b7-46944e1569d3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:44:13.821338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1e5389d6-cbf6-4a25-99b7-46944e1569d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:13.916742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 733/1495] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:44:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:44:13.921445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff649bf-3cbc-4b34-bfe4-72e28bccdbeb None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:44:13.922736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1ff649bf-3cbc-4b34-bfe4-72e28bccdbeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:13.934744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff649bf-3cbc-4b34-bfe4-72e28bccdbeb None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:44:13.998388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:44:14.032143 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c62f72-a0d9-4905-9c92-ba4d3c3946b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:44:14.033494 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c62f72-a0d9-4905-9c92-ba4d3c3946b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:14.033775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c62f72-a0d9-4905-9c92-ba4d3c3946b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:14.034042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c62f72-a0d9-4905-9c92-ba4d3c3946b4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:14.034715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c62f72-a0d9-4905-9c92-ba4d3c3946b4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:44:14.035024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c62f72-a0d9-4905-9c92-ba4d3c3946b4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:44:14.036022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: WARNING py.warnings [None req-f6c62f72-a0d9-4905-9c92-ba4d3c3946b4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:44:14.036022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: warnings.warn(msg) Aug 24 03:44:14.036022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: Aug 24 03:44:14.036919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c62f72-a0d9-4905-9c92-ba4d3c3946b4 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:44:14.048621 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 763/1496] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:44:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:44:14.056443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5389d6-cbf6-4a25-99b7-46944e1569d3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:44:14.066132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5389d6-cbf6-4a25-99b7-46944e1569d3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:44:14.127940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:44:14.162656 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888b2741-af16-43c0-9934-bbdcc865d55a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:44:14.163264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888b2741-af16-43c0-9934-bbdcc865d55a admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:14.163537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888b2741-af16-43c0-9934-bbdcc865d55a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:14.163802 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888b2741-af16-43c0-9934-bbdcc865d55a admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:14.164427 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888b2741-af16-43c0-9934-bbdcc865d55a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:44:14.166140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: WARNING py.warnings [None req-888b2741-af16-43c0-9934-bbdcc865d55a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:44:14.166140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: warnings.warn(msg) Aug 24 03:44:14.166140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: Aug 24 03:44:14.167349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888b2741-af16-43c0-9934-bbdcc865d55a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:44:14.288087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 734/1497] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:44:14 2022] POST /identity/v3/projects => generated 377 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:44:14.292406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c62f72-a0d9-4905-9c92-ba4d3c3946b4 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:44:14.293930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f6c62f72-a0d9-4905-9c92-ba4d3c3946b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:14.328154 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c62f72-a0d9-4905-9c92-ba4d3c3946b4 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:44:14.425281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b1d8a4-f9e9-4dec-bea2-86b2dfeed9e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:44:14.425876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b1d8a4-f9e9-4dec-bea2-86b2dfeed9e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:14.426175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b1d8a4-f9e9-4dec-bea2-86b2dfeed9e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:14.426457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b1d8a4-f9e9-4dec-bea2-86b2dfeed9e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:14.427092 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b1d8a4-f9e9-4dec-bea2-86b2dfeed9e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:44:14.428098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: WARNING py.warnings [None req-29b1d8a4-f9e9-4dec-bea2-86b2dfeed9e4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:44:14.428098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: warnings.warn(msg) Aug 24 03:44:14.428098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: Aug 24 03:44:14.429297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b1d8a4-f9e9-4dec-bea2-86b2dfeed9e4 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:44:14.554968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 764/1498] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:44:14 2022] POST /identity/v3/users => generated 289 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:44:14.558994 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888b2741-af16-43c0-9934-bbdcc865d55a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:44:14.560562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-888b2741-af16-43c0-9934-bbdcc865d55a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:14.622742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888b2741-af16-43c0-9934-bbdcc865d55a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:44:14.707151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:44:14.717517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4e563e-cb71-4740-8ddb-5e16784acf25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:44:14.718250 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e563e-cb71-4740-8ddb-5e16784acf25 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:14.718531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e563e-cb71-4740-8ddb-5e16784acf25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:14.718826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e563e-cb71-4740-8ddb-5e16784acf25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:14.719442 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e563e-cb71-4740-8ddb-5e16784acf25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:44:14.721068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: WARNING py.warnings [None req-3f4e563e-cb71-4740-8ddb-5e16784acf25 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:44:14.721068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: warnings.warn(msg) Aug 24 03:44:14.721068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: Aug 24 03:44:14.721949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e563e-cb71-4740-8ddb-5e16784acf25 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:44:14.735763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 735/1499] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:44:14 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:44:14.740049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b1d8a4-f9e9-4dec-bea2-86b2dfeed9e4 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:44:14.741920 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-29b1d8a4-f9e9-4dec-bea2-86b2dfeed9e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:14.754059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b1d8a4-f9e9-4dec-bea2-86b2dfeed9e4 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:44:14.779866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:44:14.849482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0811e29-8975-41c3-8afb-798aa4470f00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:44:14.850081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0811e29-8975-41c3-8afb-798aa4470f00 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:14.850386 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0811e29-8975-41c3-8afb-798aa4470f00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:14.850654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0811e29-8975-41c3-8afb-798aa4470f00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:14.851283 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0811e29-8975-41c3-8afb-798aa4470f00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:44:14.852280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: WARNING py.warnings [None req-f0811e29-8975-41c3-8afb-798aa4470f00 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:44:14.852280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: warnings.warn(msg) Aug 24 03:44:14.852280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: Aug 24 03:44:14.853165 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0811e29-8975-41c3-8afb-798aa4470f00 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:44:14.865264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 765/1500] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:44:14 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:44:14.869855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4e563e-cb71-4740-8ddb-5e16784acf25 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:44:14.871389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3f4e563e-cb71-4740-8ddb-5e16784acf25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:14.904702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4e563e-cb71-4740-8ddb-5e16784acf25 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:44:14.997368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdc5676-28f4-4d18-b97f-ca4c5c43d1ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:44:14.998020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdc5676-28f4-4d18-b97f-ca4c5c43d1ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:14.998332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdc5676-28f4-4d18-b97f-ca4c5c43d1ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:14.998602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdc5676-28f4-4d18-b97f-ca4c5c43d1ea admin admin] PATH_INFO: `/v3/projects/6ae4374949ce413aa0b58eefadac4b91/users/300d015fbfe14cd7aa3d754a5b963d79/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:15.020998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdc5676-28f4-4d18-b97f-ca4c5c43d1ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ae4374949ce413aa0b58eefadac4b91, user_id=300d015fbfe14cd7aa3d754a5b963d79, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:44:15.022019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: WARNING py.warnings [None req-8bdc5676-28f4-4d18-b97f-ca4c5c43d1ea admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:44:15.022019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: warnings.warn(msg) Aug 24 03:44:15.022019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: Aug 24 03:44:15.023987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdc5676-28f4-4d18-b97f-ca4c5c43d1ea admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:44:15.132046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 736/1501] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:44:14 2022] PUT /identity/v3/projects/6ae4374949ce413aa0b58eefadac4b91/users/300d015fbfe14cd7aa3d754a5b963d79/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:44:15.144636 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b7cff0-7e98-449b-a9d8-2a8d149e019e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:15.144922 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b7cff0-7e98-449b-a9d8-2a8d149e019e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:15.145197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b7cff0-7e98-449b-a9d8-2a8d149e019e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:15.206126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-a2b7cff0-7e98-449b-a9d8-2a8d149e019e None None] MFA Rules not processed for user `300d015fbfe14cd7aa3d754a5b963d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:44:15.207376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a2b7cff0-7e98-449b-a9d8-2a8d149e019e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:15.285382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 766/1502] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:44:15 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:44:15.299307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9057f74e-1a0f-421e-81f5-41b17469c6a8 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:15.299591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9057f74e-1a0f-421e-81f5-41b17469c6a8 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:15.299862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9057f74e-1a0f-421e-81f5-41b17469c6a8 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:15.301213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 737/1503] 10.222.0.65 () {54 vars in 964 bytes} [Wed Aug 24 03:44:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:44:15.306067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fb2c28-984c-41b8-965f-84019a376fe2 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:15.306378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fb2c28-984c-41b8-965f-84019a376fe2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:15.306654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fb2c28-984c-41b8-965f-84019a376fe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:15.337574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:44:15.385410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-64fb2c28-984c-41b8-965f-84019a376fe2 None None] MFA Rules not processed for user `dbfbfd64573141f88040692119ac5a1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:44:15.386583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-64fb2c28-984c-41b8-965f-84019a376fe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:15.463548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 767/1504] 10.222.0.65 () {58 vars in 1061 bytes} [Wed Aug 24 03:44:15 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:44:15.471626 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9057f74e-1a0f-421e-81f5-41b17469c6a8 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:44:15.473212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-9057f74e-1a0f-421e-81f5-41b17469c6a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:15.524349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:44:15.578308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9057f74e-1a0f-421e-81f5-41b17469c6a8 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:44:15.653654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f0323a-4ed2-41f0-a039-fe3449568d87 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:44:15.654205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f0323a-4ed2-41f0-a039-fe3449568d87 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:15.654491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f0323a-4ed2-41f0-a039-fe3449568d87 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:15.654764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f0323a-4ed2-41f0-a039-fe3449568d87 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:15.656381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b9f0323a-4ed2-41f0-a039-fe3449568d87 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:15.754940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f0323a-4ed2-41f0-a039-fe3449568d87 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:44:15.756380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f0323a-4ed2-41f0-a039-fe3449568d87 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:44:15.759911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:44:15.775030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 738/1505] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:44:15 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:44:15.820769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045e33a5-c0fd-4d5e-97e3-77e0f29ffe51 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:15.821189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045e33a5-c0fd-4d5e-97e3-77e0f29ffe51 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:15.821322 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045e33a5-c0fd-4d5e-97e3-77e0f29ffe51 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:15.822718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 768/1506] 10.222.0.65 () {52 vars in 830 bytes} [Wed Aug 24 03:44:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:44:15.826086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f0323a-4ed2-41f0-a039-fe3449568d87 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:44:15.827320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b9f0323a-4ed2-41f0-a039-fe3449568d87 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:15.925728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f0323a-4ed2-41f0-a039-fe3449568d87 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:44:16.017459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:44:16.018076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:16.018373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:16.018637 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] PATH_INFO: `/v3/users/300d015fbfe14cd7aa3d754a5b963d79` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:16.030676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] RBAC: Authorizing `identity:delete_user(user_id=300d015fbfe14cd7aa3d754a5b963d79)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:44:16.031807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: WARNING py.warnings [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:44:16.031807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: warnings.warn(msg) Aug 24 03:44:16.031807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: Aug 24 03:44:16.033701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:44:16.043542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:44:16.242654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '300d015fbfe14cd7aa3d754a5b963d79'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:44:16.243388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '300d015fbfe14cd7aa3d754a5b963d79'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:44:16.258123 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '300d015fbfe14cd7aa3d754a5b963d79'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:44:16.281901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '300d015fbfe14cd7aa3d754a5b963d79'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:44:16.347856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '300d015fbfe14cd7aa3d754a5b963d79'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:44:16.353482 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 739/1507] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:44:15 2022] DELETE /identity/v3/users/300d015fbfe14cd7aa3d754a5b963d79 => generated 0 bytes in 528 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:44:16.357890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1776f900-f796-4a3e-a623-518a9c1f735e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:16.365643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1776f900-f796-4a3e-a623-518a9c1f735e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:16.365643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1776f900-f796-4a3e-a623-518a9c1f735e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:16.418323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1776f900-f796-4a3e-a623-518a9c1f735e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:44:16.419554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1776f900-f796-4a3e-a623-518a9c1f735e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:16.495641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:44:16.508408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 769/1508] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:44:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:44:16.518742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:44:16.520128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:16.624502 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27470673-2760-4614-801e-3e81f2b90cba admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:44:16.685090 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:44:16.704296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca12287-3f25-46b9-b2cb-394de9a18d3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:44:16.704975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca12287-3f25-46b9-b2cb-394de9a18d3c service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:16.705251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca12287-3f25-46b9-b2cb-394de9a18d3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:16.705520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca12287-3f25-46b9-b2cb-394de9a18d3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:16.707142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-8ca12287-3f25-46b9-b2cb-394de9a18d3c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:16.818389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca12287-3f25-46b9-b2cb-394de9a18d3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:44:16.819834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca12287-3f25-46b9-b2cb-394de9a18d3c service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:44:16.839663 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 740/1509] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:44:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:44:17.304910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1776f900-f796-4a3e-a623-518a9c1f735e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:44:17.307162 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1776f900-f796-4a3e-a623-518a9c1f735e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:44:17.319397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1776f900-f796-4a3e-a623-518a9c1f735e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:44:17.423900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:44:17.424543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:44:17.424823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:44:17.425088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] PATH_INFO: `/v3/projects/6ae4374949ce413aa0b58eefadac4b91` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:44:17.435127 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ae4374949ce413aa0b58eefadac4b91)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:44:17.436153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: WARNING py.warnings [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:44:17.436153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: warnings.warn(msg) Aug 24 03:44:17.436153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: Aug 24 03:44:17.437419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:44:17.548797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ae4374949ce413aa0b58eefadac4b91', 'request_id': 'req-4310c28c-a099-4268-9215-6e64219e1479'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:44:17.555118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ae4374949ce413aa0b58eefadac4b91', 'request_id': 'req-4310c28c-a099-4268-9215-6e64219e1479'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:44:17.639586 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ae4374949ce413aa0b58eefadac4b91', 'request_id': 'req-4310c28c-a099-4268-9215-6e64219e1479'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:44:17.789988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] The token cache is being invalidate because project 6ae4374949ce413aa0b58eefadac4b91 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:44:17.790304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:44:17.790830 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4310c28c-a099-4268-9215-6e64219e1479 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:44:17.792126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 770/1510] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:44:17 2022] DELETE /identity/v3/projects/6ae4374949ce413aa0b58eefadac4b91 => generated 0 bytes in 488 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:44:26.837390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=62 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:44:27.787899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:45:59.416382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cccc0b-40bb-4193-baad-2bf8e4b85660 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:45:59.416981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cccc0b-40bb-4193-baad-2bf8e4b85660 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:45:59.416981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cccc0b-40bb-4193-baad-2bf8e4b85660 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:45:59.418301 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 741/1511] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:45:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:45:59.423438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c494a81e-d134-4bf8-8db3-99fdde5d10e3 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:45:59.423928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c494a81e-d134-4bf8-8db3-99fdde5d10e3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:45:59.424031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c494a81e-d134-4bf8-8db3-99fdde5d10e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:45:59.508868 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c494a81e-d134-4bf8-8db3-99fdde5d10e3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:45:59.510131 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c494a81e-d134-4bf8-8db3-99fdde5d10e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:45:59.618273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 771/1512] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:45:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:45:59.772488 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cccc0b-40bb-4193-baad-2bf8e4b85660 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:45:59.774674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-17cccc0b-40bb-4193-baad-2bf8e4b85660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:45:59.898126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cccc0b-40bb-4193-baad-2bf8e4b85660 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:45:59.903772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388b4994-b863-47c0-b358-5ee96c96ea67 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:45:59.904376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388b4994-b863-47c0-b358-5ee96c96ea67 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:45:59.904658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388b4994-b863-47c0-b358-5ee96c96ea67 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:45:59.904933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388b4994-b863-47c0-b358-5ee96c96ea67 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:45:59.914074 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388b4994-b863-47c0-b358-5ee96c96ea67 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:45:59.915700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388b4994-b863-47c0-b358-5ee96c96ea67 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:45:59.925492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 742/1513] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:45:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:46:00.783224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53edac9f-b2c5-41e6-a459-c1102fd860f0 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:00.783756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53edac9f-b2c5-41e6-a459-c1102fd860f0 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:00.783756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53edac9f-b2c5-41e6-a459-c1102fd860f0 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:00.785030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 772/1514] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:46:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:46:00.791165 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ced9c1-915b-4b4b-95d6-039d3084dd48 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:00.792447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ced9c1-915b-4b4b-95d6-039d3084dd48 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:00.792447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ced9c1-915b-4b4b-95d6-039d3084dd48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:00.812024 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-07ced9c1-915b-4b4b-95d6-039d3084dd48 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:46:00.813233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-07ced9c1-915b-4b4b-95d6-039d3084dd48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:00.822273 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 743/1515] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:46:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:46:00.981438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53edac9f-b2c5-41e6-a459-c1102fd860f0 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:46:00.996252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53edac9f-b2c5-41e6-a459-c1102fd860f0 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:46:01.001710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a451a8-6c18-4b35-aa6e-4ea266a6b092 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:46:01.002352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a451a8-6c18-4b35-aa6e-4ea266a6b092 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:01.002631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a451a8-6c18-4b35-aa6e-4ea266a6b092 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:01.002936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a451a8-6c18-4b35-aa6e-4ea266a6b092 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:01.012147 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a451a8-6c18-4b35-aa6e-4ea266a6b092 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:46:01.013747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a451a8-6c18-4b35-aa6e-4ea266a6b092 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:46:01.024109 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 773/1516] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:46:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:46:03.943708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b48c7a-b26a-4abf-9dc4-7f766bb655c1 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:03.975596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b48c7a-b26a-4abf-9dc4-7f766bb655c1 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:03.975596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b48c7a-b26a-4abf-9dc4-7f766bb655c1 None None] PATH_INFO: `/v3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:03.975596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 744/1517] 10.222.0.71 () {54 vars in 925 bytes} [Wed Aug 24 03:46:03 2022] GET /identity/v3 => generated 255 bytes in 4 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:46:03.975923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7436cbd8-17b2-4cfe-8796-45c3e6ddbe62 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:03.975923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7436cbd8-17b2-4cfe-8796-45c3e6ddbe62 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:03.975923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7436cbd8-17b2-4cfe-8796-45c3e6ddbe62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:04.018427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-7436cbd8-17b2-4cfe-8796-45c3e6ddbe62 None None] MFA Rules not processed for user `4b347c9c89a64168b51c439b4a92d647`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:46:04.019644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7436cbd8-17b2-4cfe-8796-45c3e6ddbe62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:04.038181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:46:04.095274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 774/1518] 10.222.0.71 () {58 vars in 1015 bytes} [Wed Aug 24 03:46:03 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:46:10.807137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=56 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:46:12.277019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b28ad19-3d0a-4e44-9d9d-dc5d0caff035 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:12.277685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b28ad19-3d0a-4e44-9d9d-dc5d0caff035 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:12.277685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b28ad19-3d0a-4e44-9d9d-dc5d0caff035 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:12.278982 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 745/1519] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:46:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:46:12.284319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6201065c-8746-4ed7-ba31-796691493edd None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:12.284762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6201065c-8746-4ed7-ba31-796691493edd None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:12.284898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6201065c-8746-4ed7-ba31-796691493edd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:12.306597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-6201065c-8746-4ed7-ba31-796691493edd None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:46:12.307824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6201065c-8746-4ed7-ba31-796691493edd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:12.316286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 775/1520] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:46:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:46:12.473652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b28ad19-3d0a-4e44-9d9d-dc5d0caff035 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:46:12.489194 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b28ad19-3d0a-4e44-9d9d-dc5d0caff035 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:46:12.496118 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91afcfc5-c035-40f9-b0b2-a888f8c4e83a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:46:12.496512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91afcfc5-c035-40f9-b0b2-a888f8c4e83a service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:12.496789 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91afcfc5-c035-40f9-b0b2-a888f8c4e83a service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:12.497059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91afcfc5-c035-40f9-b0b2-a888f8c4e83a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:12.506026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91afcfc5-c035-40f9-b0b2-a888f8c4e83a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:46:12.512780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91afcfc5-c035-40f9-b0b2-a888f8c4e83a service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:46:12.518195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 746/1521] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:46:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:46:13.353902 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f5598f-1b2b-4267-a862-e09021eb52b2 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:13.354516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f5598f-1b2b-4267-a862-e09021eb52b2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:13.354516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f5598f-1b2b-4267-a862-e09021eb52b2 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:13.355807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 776/1522] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:46:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:46:13.360831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa4af3e-10e4-4e6d-a1d7-a125b89a680a None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:13.361284 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa4af3e-10e4-4e6d-a1d7-a125b89a680a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:13.361394 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa4af3e-10e4-4e6d-a1d7-a125b89a680a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:13.382479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-6fa4af3e-10e4-4e6d-a1d7-a125b89a680a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:46:13.383690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6fa4af3e-10e4-4e6d-a1d7-a125b89a680a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:13.391759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 747/1523] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:46:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:46:13.548451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f5598f-1b2b-4267-a862-e09021eb52b2 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:46:13.564157 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f5598f-1b2b-4267-a862-e09021eb52b2 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:46:13.570361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c8c200-5261-4f94-8e3e-912f06ee4d28 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:46:13.570968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c8c200-5261-4f94-8e3e-912f06ee4d28 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:13.571234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c8c200-5261-4f94-8e3e-912f06ee4d28 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:13.571501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c8c200-5261-4f94-8e3e-912f06ee4d28 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:13.580025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c8c200-5261-4f94-8e3e-912f06ee4d28 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:46:13.581602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c8c200-5261-4f94-8e3e-912f06ee4d28 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:46:13.592004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 777/1524] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:46:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:46:16.108373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f82af17-08e8-499d-9037-50e7f27c2c5c None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:16.108823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f82af17-08e8-499d-9037-50e7f27c2c5c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:16.108929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f82af17-08e8-499d-9037-50e7f27c2c5c None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:16.110333 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 748/1525] 10.222.0.65 () {54 vars in 961 bytes} [Wed Aug 24 03:46:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:46:16.123474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b8b474-0063-48ba-9cd3-110b55c9668e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:16.123474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b8b474-0063-48ba-9cd3-110b55c9668e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:16.123474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b8b474-0063-48ba-9cd3-110b55c9668e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:16.136400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d4b8b474-0063-48ba-9cd3-110b55c9668e None None] MFA Rules not processed for user `4b347c9c89a64168b51c439b4a92d647`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:46:16.137597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d4b8b474-0063-48ba-9cd3-110b55c9668e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:16.146296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 778/1526] 10.222.0.65 () {58 vars in 1058 bytes} [Wed Aug 24 03:46:16 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:46:16.153809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f82af17-08e8-499d-9037-50e7f27c2c5c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:46:16.167691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f82af17-08e8-499d-9037-50e7f27c2c5c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:46:16.172625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975e3d2c-09d0-4774-b474-f9416c4b8aac service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:46:16.173210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975e3d2c-09d0-4774-b474-f9416c4b8aac service placement] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:16.173487 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975e3d2c-09d0-4774-b474-f9416c4b8aac service placement] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:16.173754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975e3d2c-09d0-4774-b474-f9416c4b8aac service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:16.182527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975e3d2c-09d0-4774-b474-f9416c4b8aac service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:46:16.184087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975e3d2c-09d0-4774-b474-f9416c4b8aac service placement] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:46:16.193624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 749/1527] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:46:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:46:24.818558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86784bff-f76d-4299-8d14-c69dc444b609 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:24.819108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86784bff-f76d-4299-8d14-c69dc444b609 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:24.819108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86784bff-f76d-4299-8d14-c69dc444b609 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:24.820356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 779/1528] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:46:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:46:24.825526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e24012-5196-4a7b-8ce6-55ce8481b75e None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:24.825968 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e24012-5196-4a7b-8ce6-55ce8481b75e None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:24.826072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e24012-5196-4a7b-8ce6-55ce8481b75e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:24.845942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-79e24012-5196-4a7b-8ce6-55ce8481b75e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:46:24.847190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-79e24012-5196-4a7b-8ce6-55ce8481b75e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:24.856266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 750/1529] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:46:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:46:25.013797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86784bff-f76d-4299-8d14-c69dc444b609 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:46:25.028917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86784bff-f76d-4299-8d14-c69dc444b609 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:46:25.034488 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade2c49f-f570-4a6e-9152-b41bcecd2e3d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:46:25.035094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade2c49f-f570-4a6e-9152-b41bcecd2e3d service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:25.035376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade2c49f-f570-4a6e-9152-b41bcecd2e3d service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:25.035650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade2c49f-f570-4a6e-9152-b41bcecd2e3d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:25.044773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade2c49f-f570-4a6e-9152-b41bcecd2e3d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:46:25.046412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade2c49f-f570-4a6e-9152-b41bcecd2e3d service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:46:25.056558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 780/1530] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:46:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:46:25.893431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd23eaba-3571-462a-b44c-24b1c7cbbac7 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:25.894363 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd23eaba-3571-462a-b44c-24b1c7cbbac7 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:25.894642 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd23eaba-3571-462a-b44c-24b1c7cbbac7 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:25.896520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 751/1531] 127.0.0.1 () {54 vars in 901 bytes} [Wed Aug 24 03:46:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:46:25.901809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5969a93-7c54-4f80-af04-60af93163ae5 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:25.901809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5969a93-7c54-4f80-af04-60af93163ae5 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:25.902431 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5969a93-7c54-4f80-af04-60af93163ae5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:25.925799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d5969a93-7c54-4f80-af04-60af93163ae5 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:46:25.927872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d5969a93-7c54-4f80-af04-60af93163ae5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:25.938315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 781/1532] 127.0.0.1 () {58 vars in 998 bytes} [Wed Aug 24 03:46:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:46:26.095035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd23eaba-3571-462a-b44c-24b1c7cbbac7 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:46:26.111232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd23eaba-3571-462a-b44c-24b1c7cbbac7 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:46:26.117506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521aafb5-f3dc-417a-a8fa-01968b0e922a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:46:26.118401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521aafb5-f3dc-417a-a8fa-01968b0e922a service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:26.118683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521aafb5-f3dc-417a-a8fa-01968b0e922a service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:26.118955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521aafb5-f3dc-417a-a8fa-01968b0e922a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:26.128470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521aafb5-f3dc-417a-a8fa-01968b0e922a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:46:26.130119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521aafb5-f3dc-417a-a8fa-01968b0e922a service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:46:26.141390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 752/1533] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:46:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:46:30.011425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d699d7be-e5a6-41c1-a9c3-beb789074e15 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:30.011996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d699d7be-e5a6-41c1-a9c3-beb789074e15 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:30.011996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d699d7be-e5a6-41c1-a9c3-beb789074e15 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:30.013289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 782/1534] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:46:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:46:30.019003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8690d7c6-ff31-4a48-9ff0-00f35bd19739 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:30.019445 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8690d7c6-ff31-4a48-9ff0-00f35bd19739 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:30.019578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8690d7c6-ff31-4a48-9ff0-00f35bd19739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:30.040612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-8690d7c6-ff31-4a48-9ff0-00f35bd19739 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:46:30.045097 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-8690d7c6-ff31-4a48-9ff0-00f35bd19739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:30.049847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 753/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:46:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:46:30.346767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d699d7be-e5a6-41c1-a9c3-beb789074e15 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:46:30.348467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d699d7be-e5a6-41c1-a9c3-beb789074e15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:30.428667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d699d7be-e5a6-41c1-a9c3-beb789074e15 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:46:30.432325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d227cd-2f26-4244-b771-3fa210122b4d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:46:30.432931 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d227cd-2f26-4244-b771-3fa210122b4d service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:30.433216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d227cd-2f26-4244-b771-3fa210122b4d service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:30.433486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d227cd-2f26-4244-b771-3fa210122b4d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:30.443377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d227cd-2f26-4244-b771-3fa210122b4d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:46:30.444939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d227cd-2f26-4244-b771-3fa210122b4d service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:46:30.454528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 783/1536] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:46:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:46:32.423645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9b93bc-ef4f-4b00-bad1-315f4524a611 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:32.424179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9b93bc-ef4f-4b00-bad1-315f4524a611 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:32.424179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9b93bc-ef4f-4b00-bad1-315f4524a611 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:32.425518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 754/1537] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:46:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:46:32.431158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfaa937-455d-4a35-a017-2426f7b2e21b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:32.431613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfaa937-455d-4a35-a017-2426f7b2e21b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:32.431706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfaa937-455d-4a35-a017-2426f7b2e21b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:32.452032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5dfaa937-455d-4a35-a017-2426f7b2e21b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:46:32.453243 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5dfaa937-455d-4a35-a017-2426f7b2e21b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:32.461974 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 784/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:46:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:46:32.757220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9b93bc-ef4f-4b00-bad1-315f4524a611 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:46:32.758981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7b9b93bc-ef4f-4b00-bad1-315f4524a611 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:32.772482 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9b93bc-ef4f-4b00-bad1-315f4524a611 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:46:32.776015 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25a7073-991d-4834-9fe8-161e49b46389 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:46:32.776609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a7073-991d-4834-9fe8-161e49b46389 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:32.776886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a7073-991d-4834-9fe8-161e49b46389 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:32.777154 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a7073-991d-4834-9fe8-161e49b46389 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:32.787073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25a7073-991d-4834-9fe8-161e49b46389 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:46:32.788465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25a7073-991d-4834-9fe8-161e49b46389 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:46:32.798077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 755/1539] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:46:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:46:36.368037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc0b732-8505-4029-80f9-a814ed663fe6 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:36.368683 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc0b732-8505-4029-80f9-a814ed663fe6 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:36.368683 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc0b732-8505-4029-80f9-a814ed663fe6 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:36.369889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 785/1540] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:46:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:46:36.375684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d777c2c7-4203-4687-8d9e-6e7be07a2317 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:36.376132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d777c2c7-4203-4687-8d9e-6e7be07a2317 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:36.376245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d777c2c7-4203-4687-8d9e-6e7be07a2317 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:36.396770 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d777c2c7-4203-4687-8d9e-6e7be07a2317 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:46:36.397990 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d777c2c7-4203-4687-8d9e-6e7be07a2317 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:36.407514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 756/1541] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:46:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:46:36.982317 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc0b732-8505-4029-80f9-a814ed663fe6 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:46:36.984301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6fc0b732-8505-4029-80f9-a814ed663fe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:36.988422 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=83 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:46:37.069394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc0b732-8505-4029-80f9-a814ed663fe6 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:46:37.073211 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6db6b2-4849-4d62-a2a3-feb670bd1c1d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:46:37.073829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6db6b2-4849-4d62-a2a3-feb670bd1c1d service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:37.074109 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6db6b2-4849-4d62-a2a3-feb670bd1c1d service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:37.074412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6db6b2-4849-4d62-a2a3-feb670bd1c1d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:37.086597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6db6b2-4849-4d62-a2a3-feb670bd1c1d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:46:37.087917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6db6b2-4849-4d62-a2a3-feb670bd1c1d service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:46:37.098113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 786/1542] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:46:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:46:39.092082 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39edbda1-6a17-4711-a99d-c5a74313b0b5 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:39.092660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39edbda1-6a17-4711-a99d-c5a74313b0b5 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:39.092660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39edbda1-6a17-4711-a99d-c5a74313b0b5 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:39.094001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 757/1543] 127.0.0.1 () {54 vars in 917 bytes} [Wed Aug 24 03:46:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:46:39.100190 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab1da7c-1b36-4883-bf05-a68ebe0411ed None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:39.100727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab1da7c-1b36-4883-bf05-a68ebe0411ed None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:39.100727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab1da7c-1b36-4883-bf05-a68ebe0411ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:39.120564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-eab1da7c-1b36-4883-bf05-a68ebe0411ed None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:46:39.121793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-eab1da7c-1b36-4883-bf05-a68ebe0411ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:39.130003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 787/1544] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Aug 24 03:46:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:46:39.704265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39edbda1-6a17-4711-a99d-c5a74313b0b5 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:46:39.705997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-39edbda1-6a17-4711-a99d-c5a74313b0b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:46:39.719471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39edbda1-6a17-4711-a99d-c5a74313b0b5 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:46:39.723100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2828d6-42a4-4530-84af-b8cb757d3f2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:46:39.723769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2828d6-42a4-4530-84af-b8cb757d3f2f service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:46:39.724052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2828d6-42a4-4530-84af-b8cb757d3f2f service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:46:39.724327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2828d6-42a4-4530-84af-b8cb757d3f2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:46:39.733826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2828d6-42a4-4530-84af-b8cb757d3f2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:46:39.735294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2828d6-42a4-4530-84af-b8cb757d3f2f service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:46:39.746137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 758/1545] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:46:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:46:49.114974 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=37 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:46:49.741368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=42 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:09.088607 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc658a8a-357f-4fa8-be4b-748e465db404 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.089948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc658a8a-357f-4fa8-be4b-748e465db404 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.090412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc658a8a-357f-4fa8-be4b-748e465db404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.125668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-dc658a8a-357f-4fa8-be4b-748e465db404 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:09.127357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-dc658a8a-357f-4fa8-be4b-748e465db404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:09.141548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 788/1546] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:09.150550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3972f89-78cd-4cf3-8ab7-8db319fd72e3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.151596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3972f89-78cd-4cf3-8ab7-8db319fd72e3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.152089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3972f89-78cd-4cf3-8ab7-8db319fd72e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.181934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f3972f89-78cd-4cf3-8ab7-8db319fd72e3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:09.183615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f3972f89-78cd-4cf3-8ab7-8db319fd72e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:09.199148 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 759/1547] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:09.203632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc658a8a-357f-4fa8-be4b-748e465db404 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:09.214775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc658a8a-357f-4fa8-be4b-748e465db404 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:09.224293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608a046b-456c-44d9-a05d-04235c047ab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:09.225305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a046b-456c-44d9-a05d-04235c047ab3 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.225802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a046b-456c-44d9-a05d-04235c047ab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.226338 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a046b-456c-44d9-a05d-04235c047ab3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.227981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608a046b-456c-44d9-a05d-04235c047ab3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:09.228799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608a046b-456c-44d9-a05d-04235c047ab3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:09.230812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608a046b-456c-44d9-a05d-04235c047ab3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:09.245249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 789/1548] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:48:09.253110 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3972f89-78cd-4cf3-8ab7-8db319fd72e3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:09.264254 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3972f89-78cd-4cf3-8ab7-8db319fd72e3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:09.276888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8b6011-b0d6-4845-b62e-43f1fbbe525c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:09.277930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8b6011-b0d6-4845-b62e-43f1fbbe525c admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.278459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8b6011-b0d6-4845-b62e-43f1fbbe525c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.278968 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8b6011-b0d6-4845-b62e-43f1fbbe525c admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.280152 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8b6011-b0d6-4845-b62e-43f1fbbe525c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:09.282533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8b6011-b0d6-4845-b62e-43f1fbbe525c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:09.428289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 760/1549] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:09 2022] POST /identity/v3/projects => generated 365 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:09.432740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608a046b-456c-44d9-a05d-04235c047ab3 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:09.453194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608a046b-456c-44d9-a05d-04235c047ab3 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:09.530342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96817672-ba80-4f34-884b-9277584d0f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:09.531395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96817672-ba80-4f34-884b-9277584d0f44 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.531941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96817672-ba80-4f34-884b-9277584d0f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.532437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96817672-ba80-4f34-884b-9277584d0f44 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.533323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96817672-ba80-4f34-884b-9277584d0f44 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:09.535608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96817672-ba80-4f34-884b-9277584d0f44 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:09.588472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd5010a-3de5-4cf1-93d3-c00e4a5816fd None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.588746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd5010a-3de5-4cf1-93d3-c00e4a5816fd None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.589026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd5010a-3de5-4cf1-93d3-c00e4a5816fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.624748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-afd5010a-3de5-4cf1-93d3-c00e4a5816fd None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:09.626030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-afd5010a-3de5-4cf1-93d3-c00e4a5816fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:09.644804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 761/1550] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:09.654608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c5e05-0040-4602-821b-2674af988023 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.656147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c5e05-0040-4602-821b-2674af988023 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.656147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c5e05-0040-4602-821b-2674af988023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.685269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 790/1551] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:09 2022] POST /identity/v3/users => generated 283 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:09.687647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-fd4c5e05-0040-4602-821b-2674af988023 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:09.689082 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-fd4c5e05-0040-4602-821b-2674af988023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:09.689582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcb8c79-dd6e-426d-9bb1-8e158f83bb39 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.690095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcb8c79-dd6e-426d-9bb1-8e158f83bb39 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.690635 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcb8c79-dd6e-426d-9bb1-8e158f83bb39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.703955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 762/1552] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:48:09.704905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4c5e05-0040-4602-821b-2674af988023 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:09.720877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4c5e05-0040-4602-821b-2674af988023 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:09.731007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eadc31-fda6-4390-82c0-d764852f76c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:09.731645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eadc31-fda6-4390-82c0-d764852f76c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.731926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eadc31-fda6-4390-82c0-d764852f76c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.732209 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eadc31-fda6-4390-82c0-d764852f76c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.732475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-0bcb8c79-dd6e-426d-9bb1-8e158f83bb39 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:09.732875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9eadc31-fda6-4390-82c0-d764852f76c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:09.734237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0bcb8c79-dd6e-426d-9bb1-8e158f83bb39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:09.734719 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9eadc31-fda6-4390-82c0-d764852f76c3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:09.748015 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 763/1553] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:09.748938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eadc31-fda6-4390-82c0-d764852f76c3 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:09.765595 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eadc31-fda6-4390-82c0-d764852f76c3 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:09.773594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 791/1554] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:48:09.775101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcb8c79-dd6e-426d-9bb1-8e158f83bb39 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:09.776526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b72ebc-2829-4a49-84d2-7a0f2abd391e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:09.777159 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b72ebc-2829-4a49-84d2-7a0f2abd391e admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.777438 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b72ebc-2829-4a49-84d2-7a0f2abd391e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.777710 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b72ebc-2829-4a49-84d2-7a0f2abd391e admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.778398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b72ebc-2829-4a49-84d2-7a0f2abd391e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:09.778702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b72ebc-2829-4a49-84d2-7a0f2abd391e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:09.779718 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: WARNING py.warnings [None req-20b72ebc-2829-4a49-84d2-7a0f2abd391e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:48:09.779718 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: warnings.warn(msg) Aug 24 03:48:09.779718 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: Aug 24 03:48:09.781027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b72ebc-2829-4a49-84d2-7a0f2abd391e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:09.792475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcb8c79-dd6e-426d-9bb1-8e158f83bb39 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:09.801801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 764/1555] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:48:09.802767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b23ed40-70d0-4e5e-bb13-5c09cfe7a4c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:09.803492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f972ba8a-cb0d-4d95-bd10-ef6bcd25efe3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.803775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f972ba8a-cb0d-4d95-bd10-ef6bcd25efe3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.804045 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f972ba8a-cb0d-4d95-bd10-ef6bcd25efe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.804360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b23ed40-70d0-4e5e-bb13-5c09cfe7a4c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.804849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b23ed40-70d0-4e5e-bb13-5c09cfe7a4c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.805882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b23ed40-70d0-4e5e-bb13-5c09cfe7a4c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.806800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b23ed40-70d0-4e5e-bb13-5c09cfe7a4c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:09.808652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b23ed40-70d0-4e5e-bb13-5c09cfe7a4c6 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:09.824780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 792/1556] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:09.831295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b23ed40-70d0-4e5e-bb13-5c09cfe7a4c6 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:09.851495 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f972ba8a-cb0d-4d95-bd10-ef6bcd25efe3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:09.852598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b23ed40-70d0-4e5e-bb13-5c09cfe7a4c6 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:09.854097 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f972ba8a-cb0d-4d95-bd10-ef6bcd25efe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:09.873437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9685e2a0-cc65-4302-9094-a9a9f4338e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:09.874456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9685e2a0-cc65-4302-9094-a9a9f4338e83 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.874984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9685e2a0-cc65-4302-9094-a9a9f4338e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.875461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9685e2a0-cc65-4302-9094-a9a9f4338e83 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.876300 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9685e2a0-cc65-4302-9094-a9a9f4338e83 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:09.877557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: WARNING py.warnings [None req-9685e2a0-cc65-4302-9094-a9a9f4338e83 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:48:09.877557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: warnings.warn(msg) Aug 24 03:48:09.877557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: Aug 24 03:48:09.879205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9685e2a0-cc65-4302-9094-a9a9f4338e83 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:09.879969 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 765/1557] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:09.881473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630f73a6-31b0-4c14-8464-7c6011589a63 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.881754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630f73a6-31b0-4c14-8464-7c6011589a63 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.882023 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630f73a6-31b0-4c14-8464-7c6011589a63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.925694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-630f73a6-31b0-4c14-8464-7c6011589a63 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:09.926961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-630f73a6-31b0-4c14-8464-7c6011589a63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:09.942310 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 766/1558] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:09.943167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630f73a6-31b0-4c14-8464-7c6011589a63 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:09.962630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630f73a6-31b0-4c14-8464-7c6011589a63 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:09.967744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895c1dfe-3528-44fc-a23f-34b27aeea2af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:09.971220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895c1dfe-3528-44fc-a23f-34b27aeea2af admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:09.971220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895c1dfe-3528-44fc-a23f-34b27aeea2af admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:09.971220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895c1dfe-3528-44fc-a23f-34b27aeea2af admin admin] PATH_INFO: `/v3/projects/a8fe00bed07140dc926803397fbae1bc/users/d5c12c36656a405aa9f1bad5781ecf88/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:09.987791 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895c1dfe-3528-44fc-a23f-34b27aeea2af admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8fe00bed07140dc926803397fbae1bc, user_id=d5c12c36656a405aa9f1bad5781ecf88, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:09.990764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895c1dfe-3528-44fc-a23f-34b27aeea2af admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.004330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 793/1559] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:09 2022] POST /identity/v3/projects => generated 371 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:10.005311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9685e2a0-cc65-4302-9094-a9a9f4338e83 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.068561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9685e2a0-cc65-4302-9094-a9a9f4338e83 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.076177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 767/1560] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:09 2022] PUT /identity/v3/projects/a8fe00bed07140dc926803397fbae1bc/users/d5c12c36656a405aa9f1bad5781ecf88/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:10.077608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbe7a50-6106-4bc2-8ceb-644e12b5de9d None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.077877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbe7a50-6106-4bc2-8ceb-644e12b5de9d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.078179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbe7a50-6106-4bc2-8ceb-644e12b5de9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.099447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-4cbe7a50-6106-4bc2-8ceb-644e12b5de9d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:10.100663 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4cbe7a50-6106-4bc2-8ceb-644e12b5de9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:10.116321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:10.140680 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a2fb2e-832e-4ff9-9125-3e8dfe99ed37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:10.141290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a2fb2e-832e-4ff9-9125-3e8dfe99ed37 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.141566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a2fb2e-832e-4ff9-9125-3e8dfe99ed37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.141883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a2fb2e-832e-4ff9-9125-3e8dfe99ed37 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.142567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a2fb2e-832e-4ff9-9125-3e8dfe99ed37 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:10.142863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a2fb2e-832e-4ff9-9125-3e8dfe99ed37 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:10.144546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a2fb2e-832e-4ff9-9125-3e8dfe99ed37 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.158903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 794/1561] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:48:10.163592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e32faae-5c99-4500-8d90-ccbeee9a6115 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.163892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e32faae-5c99-4500-8d90-ccbeee9a6115 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.164133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e32faae-5c99-4500-8d90-ccbeee9a6115 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.168597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 768/1562] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:10.169886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0da9e68-7793-4142-b5b5-dd29e6195042 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.170220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0da9e68-7793-4142-b5b5-dd29e6195042 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.170502 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0da9e68-7793-4142-b5b5-dd29e6195042 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.188150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-3e32faae-5c99-4500-8d90-ccbeee9a6115 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:10.189386 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3e32faae-5c99-4500-8d90-ccbeee9a6115 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:10.193616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d0da9e68-7793-4142-b5b5-dd29e6195042 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:10.194872 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d0da9e68-7793-4142-b5b5-dd29e6195042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:10.201913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 795/1563] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:10.201913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0809b23-bf20-4cc0-b48e-4c74593163e8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.201913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0809b23-bf20-4cc0-b48e-4c74593163e8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.201913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0809b23-bf20-4cc0-b48e-4c74593163e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.207000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 769/1564] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:10.208278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266600ee-8ef0-4503-9c71-04165252b823 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.208561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266600ee-8ef0-4503-9c71-04165252b823 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.208836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266600ee-8ef0-4503-9c71-04165252b823 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.221308 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-a0809b23-bf20-4cc0-b48e-4c74593163e8 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:10.222560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a0809b23-bf20-4cc0-b48e-4c74593163e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:10.229645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-266600ee-8ef0-4503-9c71-04165252b823 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:10.230895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-266600ee-8ef0-4503-9c71-04165252b823 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:10.231211 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 796/1565] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:10.232017 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0809b23-bf20-4cc0-b48e-4c74593163e8 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.241480 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 770/1566] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:10.243234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637a19e7-1527-4c51-95aa-fc5bd22a0687 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.243420 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0809b23-bf20-4cc0-b48e-4c74593163e8 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.243526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637a19e7-1527-4c51-95aa-fc5bd22a0687 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.243855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637a19e7-1527-4c51-95aa-fc5bd22a0687 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.248595 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b6fde4-3687-4594-9a87-6464086d7899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:10.249214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6fde4-3687-4594-9a87-6464086d7899 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.249576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6fde4-3687-4594-9a87-6464086d7899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.249846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6fde4-3687-4594-9a87-6464086d7899 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.250523 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b6fde4-3687-4594-9a87-6464086d7899 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:10.252462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b6fde4-3687-4594-9a87-6464086d7899 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.273510 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-637a19e7-1527-4c51-95aa-fc5bd22a0687 None None] MFA Rules not processed for user `d5c12c36656a405aa9f1bad5781ecf88`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:10.277276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-637a19e7-1527-4c51-95aa-fc5bd22a0687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:10.299974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:10.341282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 771/1567] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:10.342102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637a19e7-1527-4c51-95aa-fc5bd22a0687 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.353215 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637a19e7-1527-4c51-95aa-fc5bd22a0687 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.357887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a30998-c89e-4bf2-9c3e-e421fca01b3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:10.358510 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a30998-c89e-4bf2-9c3e-e421fca01b3c admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.358792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a30998-c89e-4bf2-9c3e-e421fca01b3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.359064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a30998-c89e-4bf2-9c3e-e421fca01b3c admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.359684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a30998-c89e-4bf2-9c3e-e421fca01b3c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:10.361592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a30998-c89e-4bf2-9c3e-e421fca01b3c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.424935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 797/1568] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/users => generated 286 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:10.426061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7428b05-4a25-43a1-a7fe-cb4d4ec4cb34 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.426440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7428b05-4a25-43a1-a7fe-cb4d4ec4cb34 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.426771 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7428b05-4a25-43a1-a7fe-cb4d4ec4cb34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.453537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c7428b05-4a25-43a1-a7fe-cb4d4ec4cb34 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:10.453537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c7428b05-4a25-43a1-a7fe-cb4d4ec4cb34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:10.456858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 798/1569] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:10.457679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7428b05-4a25-43a1-a7fe-cb4d4ec4cb34 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.468047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7428b05-4a25-43a1-a7fe-cb4d4ec4cb34 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.472785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3aabbd-a4b9-409f-b8dc-5b36c49afbde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:10.473390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3aabbd-a4b9-409f-b8dc-5b36c49afbde admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.473667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3aabbd-a4b9-409f-b8dc-5b36c49afbde admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.473938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3aabbd-a4b9-409f-b8dc-5b36c49afbde admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.474626 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3aabbd-a4b9-409f-b8dc-5b36c49afbde admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:10.474927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3aabbd-a4b9-409f-b8dc-5b36c49afbde admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:10.476537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3aabbd-a4b9-409f-b8dc-5b36c49afbde admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.489723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 799/1570] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:48:10.491051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bc3c13-a02c-4561-b15b-d2dadbafeb83 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.491339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bc3c13-a02c-4561-b15b-d2dadbafeb83 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.491614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bc3c13-a02c-4561-b15b-d2dadbafeb83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.512225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-83bc3c13-a02c-4561-b15b-d2dadbafeb83 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:10.513449 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-83bc3c13-a02c-4561-b15b-d2dadbafeb83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:10.521519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 800/1571] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:10.522776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06287216-ce5c-4404-9062-7f639acdd323 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.523060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06287216-ce5c-4404-9062-7f639acdd323 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.523359 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06287216-ce5c-4404-9062-7f639acdd323 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.540051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 772/1572] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/projects => generated 391 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:10.541523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13798d7-6f1a-4b15-ba58-a70d6756e2ff None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.542052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13798d7-6f1a-4b15-ba58-a70d6756e2ff None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.542268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13798d7-6f1a-4b15-ba58-a70d6756e2ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.544558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-06287216-ce5c-4404-9062-7f639acdd323 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:10.545876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-06287216-ce5c-4404-9062-7f639acdd323 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:10.562706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f13798d7-6f1a-4b15-ba58-a70d6756e2ff None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:10.563920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f13798d7-6f1a-4b15-ba58-a70d6756e2ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:10.611512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 801/1573] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:10.612360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06287216-ce5c-4404-9062-7f639acdd323 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.624988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06287216-ce5c-4404-9062-7f639acdd323 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.629582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11332dd5-ea6c-4f86-911b-9e7a98d869ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:10.630226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11332dd5-ea6c-4f86-911b-9e7a98d869ee admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.630510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11332dd5-ea6c-4f86-911b-9e7a98d869ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.630804 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11332dd5-ea6c-4f86-911b-9e7a98d869ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.631418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11332dd5-ea6c-4f86-911b-9e7a98d869ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:10.631556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 773/1574] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:10.632274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13798d7-6f1a-4b15-ba58-a70d6756e2ff None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.633714 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11332dd5-ea6c-4f86-911b-9e7a98d869ee admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.642349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13798d7-6f1a-4b15-ba58-a70d6756e2ff None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.647534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584f5285-eac9-4cae-b62e-32d01e57ffea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:10.648144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584f5285-eac9-4cae-b62e-32d01e57ffea admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.648423 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584f5285-eac9-4cae-b62e-32d01e57ffea admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.649239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584f5285-eac9-4cae-b62e-32d01e57ffea admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.650050 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584f5285-eac9-4cae-b62e-32d01e57ffea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:10.651779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584f5285-eac9-4cae-b62e-32d01e57ffea admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.662206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 774/1575] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:10.665102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584f5285-eac9-4cae-b62e-32d01e57ffea admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.675803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584f5285-eac9-4cae-b62e-32d01e57ffea admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.681845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890d7478-dfb2-47e8-85c1-8d73c25a093e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:10.682795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890d7478-dfb2-47e8-85c1-8d73c25a093e admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.683081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890d7478-dfb2-47e8-85c1-8d73c25a093e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.683352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890d7478-dfb2-47e8-85c1-8d73c25a093e admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.684003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890d7478-dfb2-47e8-85c1-8d73c25a093e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:10.684302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890d7478-dfb2-47e8-85c1-8d73c25a093e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:10.685962 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890d7478-dfb2-47e8-85c1-8d73c25a093e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.700674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 775/1576] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:48:10.701523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890d7478-dfb2-47e8-85c1-8d73c25a093e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.712895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890d7478-dfb2-47e8-85c1-8d73c25a093e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.719073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9974624-7604-44a6-a9d7-f5285c415077 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:10.719677 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9974624-7604-44a6-a9d7-f5285c415077 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.721893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9974624-7604-44a6-a9d7-f5285c415077 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.721893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9974624-7604-44a6-a9d7-f5285c415077 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.721893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9974624-7604-44a6-a9d7-f5285c415077 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:10.722739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9974624-7604-44a6-a9d7-f5285c415077 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.729313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 802/1577] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/projects => generated 365 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:10.730305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11332dd5-ea6c-4f86-911b-9e7a98d869ee admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.791069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:10.802457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11332dd5-ea6c-4f86-911b-9e7a98d869ee admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.807543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb11246-031a-4694-ad29-2981614ad39c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:10.808156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb11246-031a-4694-ad29-2981614ad39c admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.808438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb11246-031a-4694-ad29-2981614ad39c admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.808717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb11246-031a-4694-ad29-2981614ad39c admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.809366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb11246-031a-4694-ad29-2981614ad39c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:10.809666 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb11246-031a-4694-ad29-2981614ad39c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:10.811314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb11246-031a-4694-ad29-2981614ad39c admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.831272 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 803/1578] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:48:10.831272 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb11246-031a-4694-ad29-2981614ad39c admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.835546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb11246-031a-4694-ad29-2981614ad39c admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.841121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165049d2-450b-4ef4-998e-0baae00106b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:10.841729 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165049d2-450b-4ef4-998e-0baae00106b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.842007 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165049d2-450b-4ef4-998e-0baae00106b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.842323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165049d2-450b-4ef4-998e-0baae00106b1 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.842955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165049d2-450b-4ef4-998e-0baae00106b1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:10.844941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165049d2-450b-4ef4-998e-0baae00106b1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.889443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 776/1579] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/projects => generated 381 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:10.890851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9974624-7604-44a6-a9d7-f5285c415077 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.951010 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:10.954822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9974624-7604-44a6-a9d7-f5285c415077 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.959759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697bafee-28de-42e8-8cec-46d588f15ba8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:10.960371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697bafee-28de-42e8-8cec-46d588f15ba8 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.960650 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697bafee-28de-42e8-8cec-46d588f15ba8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.960922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697bafee-28de-42e8-8cec-46d588f15ba8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.962420 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697bafee-28de-42e8-8cec-46d588f15ba8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:10.962732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697bafee-28de-42e8-8cec-46d588f15ba8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:10.964419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697bafee-28de-42e8-8cec-46d588f15ba8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.973393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 804/1580] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:10 2022] POST /identity/v3/users => generated 296 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:10.974531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165049d2-450b-4ef4-998e-0baae00106b1 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.984772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 777/1581] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:48:10.984772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697bafee-28de-42e8-8cec-46d588f15ba8 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:10.985083 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165049d2-450b-4ef4-998e-0baae00106b1 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.989237 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697bafee-28de-42e8-8cec-46d588f15ba8 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:10.989428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1193696c-8338-4bd0-b7c8-d7a86a6ef427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:10.990048 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1193696c-8338-4bd0-b7c8-d7a86a6ef427 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:10.990362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1193696c-8338-4bd0-b7c8-d7a86a6ef427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:10.990642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1193696c-8338-4bd0-b7c8-d7a86a6ef427 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:10.991433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1193696c-8338-4bd0-b7c8-d7a86a6ef427 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:10.991737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1193696c-8338-4bd0-b7c8-d7a86a6ef427 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:10.993785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1193696c-8338-4bd0-b7c8-d7a86a6ef427 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:10.994722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87199ab5-37e4-4859-ae38-5bb39f132bd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:11.000367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87199ab5-37e4-4859-ae38-5bb39f132bd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:11.000367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87199ab5-37e4-4859-ae38-5bb39f132bd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:11.000367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87199ab5-37e4-4859-ae38-5bb39f132bd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:11.000367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87199ab5-37e4-4859-ae38-5bb39f132bd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:11.000367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87199ab5-37e4-4859-ae38-5bb39f132bd2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:11.013204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 805/1582] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:48:11.013449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 778/1583] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:11.014879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87199ab5-37e4-4859-ae38-5bb39f132bd2 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:11.017455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1193696c-8338-4bd0-b7c8-d7a86a6ef427 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:11.026678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87199ab5-37e4-4859-ae38-5bb39f132bd2 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:11.029042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1193696c-8338-4bd0-b7c8-d7a86a6ef427 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:11.031386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1839c9f4-2f82-4b6c-8cb4-01f1e2dac32c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:11.032004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1839c9f4-2f82-4b6c-8cb4-01f1e2dac32c admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:11.032279 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1839c9f4-2f82-4b6c-8cb4-01f1e2dac32c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:11.032552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1839c9f4-2f82-4b6c-8cb4-01f1e2dac32c admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:11.033172 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1839c9f4-2f82-4b6c-8cb4-01f1e2dac32c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:11.033333 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411dddb9-e70e-4672-92ae-ca8b14e8cfec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:11.033991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411dddb9-e70e-4672-92ae-ca8b14e8cfec admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:11.034203 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: WARNING py.warnings [None req-1839c9f4-2f82-4b6c-8cb4-01f1e2dac32c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:48:11.034203 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: warnings.warn(msg) Aug 24 03:48:11.034203 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: Aug 24 03:48:11.034451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411dddb9-e70e-4672-92ae-ca8b14e8cfec admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:11.034633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411dddb9-e70e-4672-92ae-ca8b14e8cfec admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:11.035182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411dddb9-e70e-4672-92ae-ca8b14e8cfec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:11.035418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1839c9f4-2f82-4b6c-8cb4-01f1e2dac32c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:11.037131 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411dddb9-e70e-4672-92ae-ca8b14e8cfec admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:11.158743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 779/1584] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:11 2022] POST /identity/v3/users => generated 289 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:11.158743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1839c9f4-2f82-4b6c-8cb4-01f1e2dac32c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:11.159091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 806/1585] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:11 2022] POST /identity/v3/projects => generated 377 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:11.159091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411dddb9-e70e-4672-92ae-ca8b14e8cfec admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:11.176821 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1839c9f4-2f82-4b6c-8cb4-01f1e2dac32c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:11.229227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411dddb9-e70e-4672-92ae-ca8b14e8cfec admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:11.234010 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3d9bae-d451-4903-9375-83a0439b0d0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:11.234654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3d9bae-d451-4903-9375-83a0439b0d0b admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:11.234931 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3d9bae-d451-4903-9375-83a0439b0d0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:11.235199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3d9bae-d451-4903-9375-83a0439b0d0b admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:11.235829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3d9bae-d451-4903-9375-83a0439b0d0b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:11.237741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3d9bae-d451-4903-9375-83a0439b0d0b admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:11.240218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36c3337-ad28-4cd8-a4c2-7394b9daded0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:11.240828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36c3337-ad28-4cd8-a4c2-7394b9daded0 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:11.241111 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36c3337-ad28-4cd8-a4c2-7394b9daded0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:11.241383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36c3337-ad28-4cd8-a4c2-7394b9daded0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:11.242041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36c3337-ad28-4cd8-a4c2-7394b9daded0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:11.244522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36c3337-ad28-4cd8-a4c2-7394b9daded0 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:11.378255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 807/1586] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:11 2022] POST /identity/v3/users => generated 291 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:11.379913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3d9bae-d451-4903-9375-83a0439b0d0b admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:11.390563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3d9bae-d451-4903-9375-83a0439b0d0b admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:11.395421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af97fead-e469-4d18-97a5-0f5454de04d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:11.396026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af97fead-e469-4d18-97a5-0f5454de04d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:11.396305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af97fead-e469-4d18-97a5-0f5454de04d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:11.396578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af97fead-e469-4d18-97a5-0f5454de04d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:11.397261 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af97fead-e469-4d18-97a5-0f5454de04d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:11.398918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af97fead-e469-4d18-97a5-0f5454de04d5 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:11.405285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 780/1587] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:11 2022] POST /identity/v3/projects => generated 373 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:11.407660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36c3337-ad28-4cd8-a4c2-7394b9daded0 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:11.414953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 808/1588] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:11 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:11.416171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af97fead-e469-4d18-97a5-0f5454de04d5 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:11.427631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af97fead-e469-4d18-97a5-0f5454de04d5 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:11.494567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36c3337-ad28-4cd8-a4c2-7394b9daded0 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:11.499413 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b2c21c-f2f7-47d0-aea3-bc6bab4dd151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:11.499590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d25718-bc4c-4af5-93de-4b2e62afe651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:11.500132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d25718-bc4c-4af5-93de-4b2e62afe651 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:11.500247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b2c21c-f2f7-47d0-aea3-bc6bab4dd151 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:11.500412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d25718-bc4c-4af5-93de-4b2e62afe651 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:11.500521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b2c21c-f2f7-47d0-aea3-bc6bab4dd151 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:11.500687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d25718-bc4c-4af5-93de-4b2e62afe651 admin admin] PATH_INFO: `/v3/projects/a180e79a3883412b8d1e24b48a7b69a4/users/67811e5632874935b3672587176d5d7d/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:11.501000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b2c21c-f2f7-47d0-aea3-bc6bab4dd151 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:11.501713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b2c21c-f2f7-47d0-aea3-bc6bab4dd151 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:11.505084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b2c21c-f2f7-47d0-aea3-bc6bab4dd151 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:11.520472 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d25718-bc4c-4af5-93de-4b2e62afe651 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a180e79a3883412b8d1e24b48a7b69a4, user_id=67811e5632874935b3672587176d5d7d, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:11.522031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: WARNING py.warnings [None req-16d25718-bc4c-4af5-93de-4b2e62afe651 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:48:11.522031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: warnings.warn(msg) Aug 24 03:48:11.522031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: Aug 24 03:48:11.524282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d25718-bc4c-4af5-93de-4b2e62afe651 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:11.527530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:11.630140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 781/1589] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:11 2022] POST /identity/v3/projects => generated 389 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:11.631046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b2c21c-f2f7-47d0-aea3-bc6bab4dd151 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:11.667536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 809/1590] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:11 2022] PUT /identity/v3/projects/a180e79a3883412b8d1e24b48a7b69a4/users/67811e5632874935b3672587176d5d7d/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:11.674181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d25718-bc4c-4af5-93de-4b2e62afe651 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:11.677111 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:11.707907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b2c21c-f2f7-47d0-aea3-bc6bab4dd151 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:11.748289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d25718-bc4c-4af5-93de-4b2e62afe651 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:11.753427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055929bc-9bb1-4aa9-a41e-69e88a41f2cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:11.754053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055929bc-9bb1-4aa9-a41e-69e88a41f2cf admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:11.754357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055929bc-9bb1-4aa9-a41e-69e88a41f2cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:11.761473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055929bc-9bb1-4aa9-a41e-69e88a41f2cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:11.761473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055929bc-9bb1-4aa9-a41e-69e88a41f2cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:11.761473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055929bc-9bb1-4aa9-a41e-69e88a41f2cf admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:11.768202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844e9485-fa4a-4feb-8fe0-a13d831fdcce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:11.768854 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844e9485-fa4a-4feb-8fe0-a13d831fdcce admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:11.769233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844e9485-fa4a-4feb-8fe0-a13d831fdcce admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:11.769534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844e9485-fa4a-4feb-8fe0-a13d831fdcce admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:11.770183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844e9485-fa4a-4feb-8fe0-a13d831fdcce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:11.770355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 810/1591] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:11 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:11.771241 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055929bc-9bb1-4aa9-a41e-69e88a41f2cf admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:11.772336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844e9485-fa4a-4feb-8fe0-a13d831fdcce admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:11.781942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055929bc-9bb1-4aa9-a41e-69e88a41f2cf admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:11.786948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeefbd5-7168-4bbf-a908-942f2e8cd5d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:11.787553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeefbd5-7168-4bbf-a908-942f2e8cd5d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:11.787832 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeefbd5-7168-4bbf-a908-942f2e8cd5d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:11.788115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeefbd5-7168-4bbf-a908-942f2e8cd5d0 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:11.788739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aeefbd5-7168-4bbf-a908-942f2e8cd5d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:11.791499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aeefbd5-7168-4bbf-a908-942f2e8cd5d0 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:11.905947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 782/1592] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:11 2022] POST /identity/v3/projects => generated 385 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:11.906856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844e9485-fa4a-4feb-8fe0-a13d831fdcce admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:11.938400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 811/1593] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:11 2022] POST /identity/v3/users => generated 289 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:11.944871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeefbd5-7168-4bbf-a908-942f2e8cd5d0 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:11.949587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeefbd5-7168-4bbf-a908-942f2e8cd5d0 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:11.977961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844e9485-fa4a-4feb-8fe0-a13d831fdcce admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:11.983023 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bb3f5a-d961-48c2-819f-632c032e88ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:11.983627 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bb3f5a-d961-48c2-819f-632c032e88ab admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:11.983902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bb3f5a-d961-48c2-819f-632c032e88ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:11.984171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bb3f5a-d961-48c2-819f-632c032e88ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:11.985613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bb3f5a-d961-48c2-819f-632c032e88ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:11.986528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bb3f5a-d961-48c2-819f-632c032e88ab admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:11.997613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 783/1594] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:11 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:11.998621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bb3f5a-d961-48c2-819f-632c032e88ab admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.008047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bb3f5a-d961-48c2-819f-632c032e88ab admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.011307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f16888b-ac42-45ab-9473-529d30f108be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.011925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f16888b-ac42-45ab-9473-529d30f108be admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.012204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f16888b-ac42-45ab-9473-529d30f108be admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.012480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f16888b-ac42-45ab-9473-529d30f108be admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.013105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f16888b-ac42-45ab-9473-529d30f108be admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.013258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bca136-2406-4d3f-88b2-931dae34da6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.013807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bca136-2406-4d3f-88b2-931dae34da6c admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.014082 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bca136-2406-4d3f-88b2-931dae34da6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.014384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bca136-2406-4d3f-88b2-931dae34da6c admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.015091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bca136-2406-4d3f-88b2-931dae34da6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.015244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f16888b-ac42-45ab-9473-529d30f108be admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.016661 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bca136-2406-4d3f-88b2-931dae34da6c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.027934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 784/1595] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:12.028852 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bca136-2406-4d3f-88b2-931dae34da6c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.038550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bca136-2406-4d3f-88b2-931dae34da6c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.044047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e092e03-a9bc-4690-a17c-78c6955110d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.044620 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e092e03-a9bc-4690-a17c-78c6955110d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.044899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e092e03-a9bc-4690-a17c-78c6955110d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.045172 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e092e03-a9bc-4690-a17c-78c6955110d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.045780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e092e03-a9bc-4690-a17c-78c6955110d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.047716 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e092e03-a9bc-4690-a17c-78c6955110d8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.138739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 812/1596] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:11 2022] POST /identity/v3/users => generated 287 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:12.140156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c733cd5-d95d-4817-a5ed-c63b6060b84a None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.140440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c733cd5-d95d-4817-a5ed-c63b6060b84a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.140716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c733cd5-d95d-4817-a5ed-c63b6060b84a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.170427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-3c733cd5-d95d-4817-a5ed-c63b6060b84a None None] MFA Rules not processed for user `67811e5632874935b3672587176d5d7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:12.171657 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3c733cd5-d95d-4817-a5ed-c63b6060b84a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:12.187011 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 785/1597] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:12 2022] POST /identity/v3/users => generated 295 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:12.188588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e092e03-a9bc-4690-a17c-78c6955110d8 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.202297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e092e03-a9bc-4690-a17c-78c6955110d8 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.208315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c53a569-b952-4ccd-a0e7-013f9bb62f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.209177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c53a569-b952-4ccd-a0e7-013f9bb62f2d admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.209592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c53a569-b952-4ccd-a0e7-013f9bb62f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.209955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c53a569-b952-4ccd-a0e7-013f9bb62f2d admin admin] PATH_INFO: `/v3/projects/eb3f65c2c4b2464893d90fbab9dd7968/users/651b943399c34c0f933ea32b7e1ed065/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.228038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c53a569-b952-4ccd-a0e7-013f9bb62f2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb3f65c2c4b2464893d90fbab9dd7968, user_id=651b943399c34c0f933ea32b7e1ed065, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.230764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c53a569-b952-4ccd-a0e7-013f9bb62f2d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.236456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:12.246872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 813/1598] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:12 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:12.247703 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c733cd5-d95d-4817-a5ed-c63b6060b84a None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.258763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c733cd5-d95d-4817-a5ed-c63b6060b84a None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.263326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518b322a-c54f-4b67-9b60-458eba29f3dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.263946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518b322a-c54f-4b67-9b60-458eba29f3dc admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.264223 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518b322a-c54f-4b67-9b60-458eba29f3dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.264489 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518b322a-c54f-4b67-9b60-458eba29f3dc admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.265121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518b322a-c54f-4b67-9b60-458eba29f3dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.267101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518b322a-c54f-4b67-9b60-458eba29f3dc admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.298420 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 786/1599] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:12 2022] PUT /identity/v3/projects/eb3f65c2c4b2464893d90fbab9dd7968/users/651b943399c34c0f933ea32b7e1ed065/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:12.299360 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c53a569-b952-4ccd-a0e7-013f9bb62f2d admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.309534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:12.362368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c53a569-b952-4ccd-a0e7-013f9bb62f2d admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.366962 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86cac59-332d-48ba-9bfc-366740e0d6ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.368292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86cac59-332d-48ba-9bfc-366740e0d6ca admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.368566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86cac59-332d-48ba-9bfc-366740e0d6ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.368834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86cac59-332d-48ba-9bfc-366740e0d6ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.369476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86cac59-332d-48ba-9bfc-366740e0d6ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.371105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86cac59-332d-48ba-9bfc-366740e0d6ca admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.385004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 787/1600] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:12.385004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86cac59-332d-48ba-9bfc-366740e0d6ca admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.385321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 814/1601] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:12 2022] POST /identity/v3/users => generated 293 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:12.385321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518b322a-c54f-4b67-9b60-458eba29f3dc admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.394749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86cac59-332d-48ba-9bfc-366740e0d6ca admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.395379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518b322a-c54f-4b67-9b60-458eba29f3dc admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.400243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b3ac65-83c9-4eb9-894c-f138c1b11c79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.400863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b3ac65-83c9-4eb9-894c-f138c1b11c79 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.401140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b3ac65-83c9-4eb9-894c-f138c1b11c79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.401411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b3ac65-83c9-4eb9-894c-f138c1b11c79 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.402060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b3ac65-83c9-4eb9-894c-f138c1b11c79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.402197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfea86e-0588-4765-8ca2-e8c36500ba80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.403192 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfea86e-0588-4765-8ca2-e8c36500ba80 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.403671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfea86e-0588-4765-8ca2-e8c36500ba80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.403840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b3ac65-83c9-4eb9-894c-f138c1b11c79 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.404228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfea86e-0588-4765-8ca2-e8c36500ba80 admin admin] PATH_INFO: `/v3/projects/c993aed86aaf431495a2d1724218fa9b/users/0772f8b4306f4e239f075f7b1c13790f/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.419267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 788/1602] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:12.419267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b3ac65-83c9-4eb9-894c-f138c1b11c79 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.428357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfea86e-0588-4765-8ca2-e8c36500ba80 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c993aed86aaf431495a2d1724218fa9b, user_id=0772f8b4306f4e239f075f7b1c13790f, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.430914 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b3ac65-83c9-4eb9-894c-f138c1b11c79 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.431173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfea86e-0588-4765-8ca2-e8c36500ba80 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.435984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0888339-33eb-445e-b51a-6b7cd224f57c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.436596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0888339-33eb-445e-b51a-6b7cd224f57c admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.436873 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0888339-33eb-445e-b51a-6b7cd224f57c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.437143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0888339-33eb-445e-b51a-6b7cd224f57c admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.437804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0888339-33eb-445e-b51a-6b7cd224f57c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.439518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0888339-33eb-445e-b51a-6b7cd224f57c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.452144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 789/1603] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:12.453073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0888339-33eb-445e-b51a-6b7cd224f57c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.464368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0888339-33eb-445e-b51a-6b7cd224f57c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.471264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26a09dd-4d0a-4584-9f96-a878f0e7b6fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.472198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26a09dd-4d0a-4584-9f96-a878f0e7b6fd admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.472483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26a09dd-4d0a-4584-9f96-a878f0e7b6fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.472809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26a09dd-4d0a-4584-9f96-a878f0e7b6fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.473479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26a09dd-4d0a-4584-9f96-a878f0e7b6fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.475185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26a09dd-4d0a-4584-9f96-a878f0e7b6fd admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.486401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 790/1604] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:12.487374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26a09dd-4d0a-4584-9f96-a878f0e7b6fd admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.497959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26a09dd-4d0a-4584-9f96-a878f0e7b6fd admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.502930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e67358-7cb7-4b94-8df1-f78f2cb58c61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.503521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e67358-7cb7-4b94-8df1-f78f2cb58c61 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.503798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e67358-7cb7-4b94-8df1-f78f2cb58c61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.504066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e67358-7cb7-4b94-8df1-f78f2cb58c61 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.504691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e67358-7cb7-4b94-8df1-f78f2cb58c61 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.506629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e67358-7cb7-4b94-8df1-f78f2cb58c61 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.522380 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 815/1605] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:12 2022] PUT /identity/v3/projects/c993aed86aaf431495a2d1724218fa9b/users/0772f8b4306f4e239f075f7b1c13790f/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:12.523525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfea86e-0588-4765-8ca2-e8c36500ba80 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.611618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfea86e-0588-4765-8ca2-e8c36500ba80 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.619291 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714d4627-93fc-4967-86c2-87d0aa06100c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.620255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714d4627-93fc-4967-86c2-87d0aa06100c admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.620723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714d4627-93fc-4967-86c2-87d0aa06100c admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.621160 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714d4627-93fc-4967-86c2-87d0aa06100c admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.622042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714d4627-93fc-4967-86c2-87d0aa06100c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.623704 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714d4627-93fc-4967-86c2-87d0aa06100c admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.631138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 791/1606] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:12 2022] POST /identity/v3/projects => generated 371 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:12.632066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e67358-7cb7-4b94-8df1-f78f2cb58c61 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.636207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 816/1607] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:12.640226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714d4627-93fc-4967-86c2-87d0aa06100c admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.650824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714d4627-93fc-4967-86c2-87d0aa06100c admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.701840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e67358-7cb7-4b94-8df1-f78f2cb58c61 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.706952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6a1281-2581-402e-bb99-6da3c6acdab8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.707627 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6a1281-2581-402e-bb99-6da3c6acdab8 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.707917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6a1281-2581-402e-bb99-6da3c6acdab8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.708187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6a1281-2581-402e-bb99-6da3c6acdab8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.708837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6a1281-2581-402e-bb99-6da3c6acdab8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.710474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6a1281-2581-402e-bb99-6da3c6acdab8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.711530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a24fa86-d1d9-45e7-990f-1e0d4fd87f52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.712316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24fa86-d1d9-45e7-990f-1e0d4fd87f52 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.712823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24fa86-d1d9-45e7-990f-1e0d4fd87f52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.713105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24fa86-d1d9-45e7-990f-1e0d4fd87f52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.714234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a24fa86-d1d9-45e7-990f-1e0d4fd87f52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.715870 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a24fa86-d1d9-45e7-990f-1e0d4fd87f52 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.725531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 792/1608] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:12.725531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6a1281-2581-402e-bb99-6da3c6acdab8 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.728173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 817/1609] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:12.728173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a24fa86-d1d9-45e7-990f-1e0d4fd87f52 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.737265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6a1281-2581-402e-bb99-6da3c6acdab8 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.741699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a24fa86-d1d9-45e7-990f-1e0d4fd87f52 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.742073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9529a6a4-1965-47bf-94ae-9d6daa1bfb68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.743356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9529a6a4-1965-47bf-94ae-9d6daa1bfb68 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.743637 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9529a6a4-1965-47bf-94ae-9d6daa1bfb68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.743915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9529a6a4-1965-47bf-94ae-9d6daa1bfb68 admin admin] PATH_INFO: `/v3/projects/8f7520af1ae14ad28eec126353eaac53/users/77b4b56f008c44c081130377cf8ce630/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.747751 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc5d3fa-d602-4a9b-90dc-722fe6ff477b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.748363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc5d3fa-d602-4a9b-90dc-722fe6ff477b admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.748643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc5d3fa-d602-4a9b-90dc-722fe6ff477b admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.748915 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc5d3fa-d602-4a9b-90dc-722fe6ff477b admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.749589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc5d3fa-d602-4a9b-90dc-722fe6ff477b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.751288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc5d3fa-d602-4a9b-90dc-722fe6ff477b admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.760674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9529a6a4-1965-47bf-94ae-9d6daa1bfb68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f7520af1ae14ad28eec126353eaac53, user_id=77b4b56f008c44c081130377cf8ce630, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.763377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9529a6a4-1965-47bf-94ae-9d6daa1bfb68 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.763519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 818/1610] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:12.764338 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc5d3fa-d602-4a9b-90dc-722fe6ff477b admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.780275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc5d3fa-d602-4a9b-90dc-722fe6ff477b admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.783438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019920b1-41cb-4800-9cf8-227d20ad9709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.784038 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019920b1-41cb-4800-9cf8-227d20ad9709 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.784315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019920b1-41cb-4800-9cf8-227d20ad9709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.784583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019920b1-41cb-4800-9cf8-227d20ad9709 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.785218 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019920b1-41cb-4800-9cf8-227d20ad9709 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.786880 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019920b1-41cb-4800-9cf8-227d20ad9709 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.797507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 819/1611] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:12 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:12.798750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0990c14f-e429-4750-abf8-89f58988ea1e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.799029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0990c14f-e429-4750-abf8-89f58988ea1e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.799297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0990c14f-e429-4750-abf8-89f58988ea1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.818627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-0990c14f-e429-4750-abf8-89f58988ea1e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:12.819898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0990c14f-e429-4750-abf8-89f58988ea1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:12.828510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 820/1612] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:12.829301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0990c14f-e429-4750-abf8-89f58988ea1e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.840528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0990c14f-e429-4750-abf8-89f58988ea1e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.843759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 793/1613] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:12 2022] PUT /identity/v3/projects/8f7520af1ae14ad28eec126353eaac53/users/77b4b56f008c44c081130377cf8ce630/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:12.844705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9529a6a4-1965-47bf-94ae-9d6daa1bfb68 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:12.858789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:12.873268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:12.907340 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f160d90-a1da-4064-bb44-09e8270efd3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.907952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f160d90-a1da-4064-bb44-09e8270efd3d admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.908225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f160d90-a1da-4064-bb44-09e8270efd3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.908492 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f160d90-a1da-4064-bb44-09e8270efd3d admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.909126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f160d90-a1da-4064-bb44-09e8270efd3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.911115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f160d90-a1da-4064-bb44-09e8270efd3d admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.912866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9529a6a4-1965-47bf-94ae-9d6daa1bfb68 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:12.917864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024bcd06-982f-4f11-a034-db4b20da0dd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:12.918563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024bcd06-982f-4f11-a034-db4b20da0dd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:12.919048 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024bcd06-982f-4f11-a034-db4b20da0dd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:12.919248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024bcd06-982f-4f11-a034-db4b20da0dd9 admin admin] PATH_INFO: `/v3/projects/eb3f65c2c4b2464893d90fbab9dd7968/users/651b943399c34c0f933ea32b7e1ed065/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:12.921351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024bcd06-982f-4f11-a034-db4b20da0dd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb3f65c2c4b2464893d90fbab9dd7968, user_id=651b943399c34c0f933ea32b7e1ed065, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:12.924037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024bcd06-982f-4f11-a034-db4b20da0dd9 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:12.936897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 794/1614] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:12 2022] PUT /identity/v3/projects/eb3f65c2c4b2464893d90fbab9dd7968/users/651b943399c34c0f933ea32b7e1ed065/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:12.937894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024bcd06-982f-4f11-a034-db4b20da0dd9 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:13.002105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024bcd06-982f-4f11-a034-db4b20da0dd9 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:13.006937 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24caed24-fb6a-4daa-aea2-e0d392278923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:13.007600 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24caed24-fb6a-4daa-aea2-e0d392278923 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.007916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24caed24-fb6a-4daa-aea2-e0d392278923 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.008189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24caed24-fb6a-4daa-aea2-e0d392278923 admin admin] PATH_INFO: `/v3/projects/dacde99115e641439164fc91e75f2aa0/users/728c3f2735d6441b870cf146fbf084c0/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.018253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 821/1615] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:12 2022] POST /identity/v3/users => generated 292 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:13.019333 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f160d90-a1da-4064-bb44-09e8270efd3d admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:13.023730 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24caed24-fb6a-4daa-aea2-e0d392278923 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dacde99115e641439164fc91e75f2aa0, user_id=728c3f2735d6441b870cf146fbf084c0, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:13.026697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24caed24-fb6a-4daa-aea2-e0d392278923 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:13.030462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f160d90-a1da-4064-bb44-09e8270efd3d admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:13.035205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b72d19-2ce9-44c0-b021-5d6fa630c195 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:13.035807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b72d19-2ce9-44c0-b021-5d6fa630c195 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.036084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b72d19-2ce9-44c0-b021-5d6fa630c195 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.036354 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b72d19-2ce9-44c0-b021-5d6fa630c195 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.037001 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b72d19-2ce9-44c0-b021-5d6fa630c195 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:13.038638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b72d19-2ce9-44c0-b021-5d6fa630c195 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:13.049799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 822/1616] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:13 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:13.051320 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b72d19-2ce9-44c0-b021-5d6fa630c195 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:13.061806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b72d19-2ce9-44c0-b021-5d6fa630c195 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:13.066575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e9dc54-212c-46c1-8000-88ef03e66741 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:13.067227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e9dc54-212c-46c1-8000-88ef03e66741 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.067505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e9dc54-212c-46c1-8000-88ef03e66741 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.067816 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e9dc54-212c-46c1-8000-88ef03e66741 admin admin] PATH_INFO: `/v3/projects/dbaf4ab3fd764ce88e7af0dbc3e6601e/users/7f4991f9c6ca4fc99f756f75f9460808/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.083124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e9dc54-212c-46c1-8000-88ef03e66741 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbaf4ab3fd764ce88e7af0dbc3e6601e, user_id=7f4991f9c6ca4fc99f756f75f9460808, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:13.085792 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e9dc54-212c-46c1-8000-88ef03e66741 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:13.110363 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 795/1617] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:12 2022] PUT /identity/v3/projects/dacde99115e641439164fc91e75f2aa0/users/728c3f2735d6441b870cf146fbf084c0/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:13.111272 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24caed24-fb6a-4daa-aea2-e0d392278923 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:13.180453 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24caed24-fb6a-4daa-aea2-e0d392278923 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:13.185468 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6138a37-4a8f-4db9-8515-888beec9956f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:13.186134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6138a37-4a8f-4db9-8515-888beec9956f admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.186586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6138a37-4a8f-4db9-8515-888beec9956f admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.186772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6138a37-4a8f-4db9-8515-888beec9956f admin admin] PATH_INFO: `/v3/projects/ffc635aa12d14aa0b2880abe4e4f8f2f/users/bd8d967d9ecb410ea47e130ac3c8b4a6/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.202518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6138a37-4a8f-4db9-8515-888beec9956f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffc635aa12d14aa0b2880abe4e4f8f2f, user_id=bd8d967d9ecb410ea47e130ac3c8b4a6, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:13.205311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6138a37-4a8f-4db9-8515-888beec9956f admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:13.207504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 823/1618] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:13 2022] PUT /identity/v3/projects/dbaf4ab3fd764ce88e7af0dbc3e6601e/users/7f4991f9c6ca4fc99f756f75f9460808/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:13.208413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e9dc54-212c-46c1-8000-88ef03e66741 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:13.270301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e9dc54-212c-46c1-8000-88ef03e66741 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:13.274461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1b18ec-4515-4981-b41f-1fdbb7783202 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:13.275067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1b18ec-4515-4981-b41f-1fdbb7783202 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.275344 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1b18ec-4515-4981-b41f-1fdbb7783202 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.275612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1b18ec-4515-4981-b41f-1fdbb7783202 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.340129 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 796/1619] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:13 2022] PUT /identity/v3/projects/ffc635aa12d14aa0b2880abe4e4f8f2f/users/bd8d967d9ecb410ea47e130ac3c8b4a6/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:13.341416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ee51be-05f9-4256-95f9-2b21edc01a12 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.341707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ee51be-05f9-4256-95f9-2b21edc01a12 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.341965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ee51be-05f9-4256-95f9-2b21edc01a12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.346453 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1b18ec-4515-4981-b41f-1fdbb7783202 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:13.347926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1b18ec-4515-4981-b41f-1fdbb7783202 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:13.359277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 824/1620] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:13.360540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd0a53b-dc29-44d3-8103-50dca5a48d70 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.360803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd0a53b-dc29-44d3-8103-50dca5a48d70 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.361068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd0a53b-dc29-44d3-8103-50dca5a48d70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.362471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-54ee51be-05f9-4256-95f9-2b21edc01a12 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:13.363713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-54ee51be-05f9-4256-95f9-2b21edc01a12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:13.384662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:13.391008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-fdd0a53b-dc29-44d3-8103-50dca5a48d70 None None] MFA Rules not processed for user `651b943399c34c0f933ea32b7e1ed065`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:13.392250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-fdd0a53b-dc29-44d3-8103-50dca5a48d70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:13.401162 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:13.431070 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 797/1621] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:13.431997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ee51be-05f9-4256-95f9-2b21edc01a12 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:13.441444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ee51be-05f9-4256-95f9-2b21edc01a12 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:13.446685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d77ded-6040-450d-9b0c-2a0d9e98de09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:13.447300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d77ded-6040-450d-9b0c-2a0d9e98de09 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.447628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d77ded-6040-450d-9b0c-2a0d9e98de09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.447899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d77ded-6040-450d-9b0c-2a0d9e98de09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.448549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d77ded-6040-450d-9b0c-2a0d9e98de09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:13.450210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d77ded-6040-450d-9b0c-2a0d9e98de09 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:13.464276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 798/1622] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:13 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:13.464276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d77ded-6040-450d-9b0c-2a0d9e98de09 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:13.475300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d77ded-6040-450d-9b0c-2a0d9e98de09 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:13.476116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 825/1623] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:13 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:13.478422 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ccb93e-af89-4d1a-bf12-99d9a1ddb7f2 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.478911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ccb93e-af89-4d1a-bf12-99d9a1ddb7f2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.479357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ccb93e-af89-4d1a-bf12-99d9a1ddb7f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.481375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607c5a92-bcdf-4483-b747-6e36f0d5aebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:13.482143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607c5a92-bcdf-4483-b747-6e36f0d5aebd admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.482651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607c5a92-bcdf-4483-b747-6e36f0d5aebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.482906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607c5a92-bcdf-4483-b747-6e36f0d5aebd admin admin] PATH_INFO: `/v3/projects/6252f05139f14716a83b4ea1cd7048ad/users/28150f2ee18e47c783f906a162e568bb/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.501156 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607c5a92-bcdf-4483-b747-6e36f0d5aebd admin admin] RBAC: Authorizing `identity:create_grant(project_id=6252f05139f14716a83b4ea1cd7048ad, user_id=28150f2ee18e47c783f906a162e568bb, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:13.504092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607c5a92-bcdf-4483-b747-6e36f0d5aebd admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:13.504629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-22ccb93e-af89-4d1a-bf12-99d9a1ddb7f2 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:13.505853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-22ccb93e-af89-4d1a-bf12-99d9a1ddb7f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:13.514512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 826/1624] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:13.515915 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413fbbf3-aec2-461e-b586-62935f01469f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.516197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413fbbf3-aec2-461e-b586-62935f01469f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.516490 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413fbbf3-aec2-461e-b586-62935f01469f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.536965 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-413fbbf3-aec2-461e-b586-62935f01469f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:13.538228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-413fbbf3-aec2-461e-b586-62935f01469f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:13.546832 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 827/1625] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:13.548184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f103d5-31db-418f-88ae-93a7e8d86984 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.548468 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f103d5-31db-418f-88ae-93a7e8d86984 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.548738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f103d5-31db-418f-88ae-93a7e8d86984 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.567751 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-94f103d5-31db-418f-88ae-93a7e8d86984 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:13.568969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-94f103d5-31db-418f-88ae-93a7e8d86984 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:13.577028 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 828/1626] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:13.577978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f103d5-31db-418f-88ae-93a7e8d86984 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:13.634823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f103d5-31db-418f-88ae-93a7e8d86984 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:13.636128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 799/1627] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:13 2022] PUT /identity/v3/projects/6252f05139f14716a83b4ea1cd7048ad/users/28150f2ee18e47c783f906a162e568bb/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:13.636739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607c5a92-bcdf-4483-b747-6e36f0d5aebd admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:13.710440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a12e570-4df1-4f8d-80bb-c1204681c8d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:13.712100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a12e570-4df1-4f8d-80bb-c1204681c8d0 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.712435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a12e570-4df1-4f8d-80bb-c1204681c8d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.712711 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a12e570-4df1-4f8d-80bb-c1204681c8d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.714457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607c5a92-bcdf-4483-b747-6e36f0d5aebd admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:13.719599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d19eb0-3946-4f86-9765-03a07b2aba6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:13.720334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d19eb0-3946-4f86-9765-03a07b2aba6d admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.720688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d19eb0-3946-4f86-9765-03a07b2aba6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.721046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d19eb0-3946-4f86-9765-03a07b2aba6d admin admin] PATH_INFO: `/v3/projects/0030b901e26545a5a136cbcbbab21b5e/users/462945d4cfad48668268e66dd7f835e5/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.722931 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a12e570-4df1-4f8d-80bb-c1204681c8d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:13.725721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a12e570-4df1-4f8d-80bb-c1204681c8d0 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:13.737786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 829/1628] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:13.741798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a12e570-4df1-4f8d-80bb-c1204681c8d0 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:13.741917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d19eb0-3946-4f86-9765-03a07b2aba6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=0030b901e26545a5a136cbcbbab21b5e, user_id=462945d4cfad48668268e66dd7f835e5, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:13.742881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d19eb0-3946-4f86-9765-03a07b2aba6d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:13.794823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:13.803867 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a12e570-4df1-4f8d-80bb-c1204681c8d0 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:13.808108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e372a4c-dfdb-432a-b5c1-ae8813932f3a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:13.808725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e372a4c-dfdb-432a-b5c1-ae8813932f3a service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.809011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e372a4c-dfdb-432a-b5c1-ae8813932f3a service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.809285 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e372a4c-dfdb-432a-b5c1-ae8813932f3a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.835677 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 800/1629] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:13 2022] PUT /identity/v3/projects/0030b901e26545a5a136cbcbbab21b5e/users/462945d4cfad48668268e66dd7f835e5/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:13.836608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d19eb0-3946-4f86-9765-03a07b2aba6d admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:13.883516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e372a4c-dfdb-432a-b5c1-ae8813932f3a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:13.885451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e372a4c-dfdb-432a-b5c1-ae8813932f3a service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:13.897868 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 830/1630] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:13 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:13.899653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e372a4c-dfdb-432a-b5c1-ae8813932f3a service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:13.904934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d19eb0-3946-4f86-9765-03a07b2aba6d admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:13.910052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e372a4c-dfdb-432a-b5c1-ae8813932f3a service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:13.911268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b996224e-e2ee-4c07-8c82-5f86baf35284 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:13.912515 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b996224e-e2ee-4c07-8c82-5f86baf35284 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.913089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b996224e-e2ee-4c07-8c82-5f86baf35284 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.913722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b996224e-e2ee-4c07-8c82-5f86baf35284 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.916144 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299c10cb-4c3f-453e-8da0-bd5bbaf6477a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:13.916938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299c10cb-4c3f-453e-8da0-bd5bbaf6477a service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:13.917292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299c10cb-4c3f-453e-8da0-bd5bbaf6477a service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:13.917627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299c10cb-4c3f-453e-8da0-bd5bbaf6477a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:13.956531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:14.022262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b996224e-e2ee-4c07-8c82-5f86baf35284 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:14.022262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b996224e-e2ee-4c07-8c82-5f86baf35284 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:14.022460 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299c10cb-4c3f-453e-8da0-bd5bbaf6477a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:14.022460 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299c10cb-4c3f-453e-8da0-bd5bbaf6477a service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:14.042808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 801/1631] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:14.043824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b996224e-e2ee-4c07-8c82-5f86baf35284 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:14.047020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 831/1632] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:14.048340 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94926086-e3f4-4780-b06b-02f8efe09a98 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:14.048617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94926086-e3f4-4780-b06b-02f8efe09a98 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:14.048884 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94926086-e3f4-4780-b06b-02f8efe09a98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:14.054609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b996224e-e2ee-4c07-8c82-5f86baf35284 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:14.061581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606becaa-c8ff-4fb5-bde3-e13ef5d77964 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:14.061581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606becaa-c8ff-4fb5-bde3-e13ef5d77964 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:14.061581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606becaa-c8ff-4fb5-bde3-e13ef5d77964 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:14.061581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606becaa-c8ff-4fb5-bde3-e13ef5d77964 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:14.082855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606becaa-c8ff-4fb5-bde3-e13ef5d77964 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:14.084372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606becaa-c8ff-4fb5-bde3-e13ef5d77964 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:14.097042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-94926086-e3f4-4780-b06b-02f8efe09a98 None None] MFA Rules not processed for user `28150f2ee18e47c783f906a162e568bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:14.102035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-94926086-e3f4-4780-b06b-02f8efe09a98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:14.102231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 802/1633] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:14.102231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606becaa-c8ff-4fb5-bde3-e13ef5d77964 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:14.110948 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606becaa-c8ff-4fb5-bde3-e13ef5d77964 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:14.118338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39420e3d-5eea-404d-9cdc-276e3a3d9ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:14.119564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39420e3d-5eea-404d-9cdc-276e3a3d9ec0 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:14.119564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39420e3d-5eea-404d-9cdc-276e3a3d9ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:14.119847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39420e3d-5eea-404d-9cdc-276e3a3d9ec0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:14.120623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39420e3d-5eea-404d-9cdc-276e3a3d9ec0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:14.122661 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39420e3d-5eea-404d-9cdc-276e3a3d9ec0 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:14.139308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 803/1634] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:14 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:14.140561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39420e3d-5eea-404d-9cdc-276e3a3d9ec0 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:14.156113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39420e3d-5eea-404d-9cdc-276e3a3d9ec0 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:14.165597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e35829-d42a-4059-9cce-fc027ddf8362 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:14.167031 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e35829-d42a-4059-9cce-fc027ddf8362 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:14.167618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e35829-d42a-4059-9cce-fc027ddf8362 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:14.167995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e35829-d42a-4059-9cce-fc027ddf8362 admin admin] PATH_INFO: `/v3/projects/0030b901e26545a5a136cbcbbab21b5e/users/462945d4cfad48668268e66dd7f835e5/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:14.170764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e35829-d42a-4059-9cce-fc027ddf8362 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0030b901e26545a5a136cbcbbab21b5e, user_id=462945d4cfad48668268e66dd7f835e5, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:14.174562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e35829-d42a-4059-9cce-fc027ddf8362 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:14.202659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 804/1635] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:14 2022] PUT /identity/v3/projects/0030b901e26545a5a136cbcbbab21b5e/users/462945d4cfad48668268e66dd7f835e5/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:14.222518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b523f7-6d28-470e-b504-2c43a7c7ccb4 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:14.223597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b523f7-6d28-470e-b504-2c43a7c7ccb4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:14.224264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b523f7-6d28-470e-b504-2c43a7c7ccb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:14.260568 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-68b523f7-6d28-470e-b504-2c43a7c7ccb4 None None] MFA Rules not processed for user `462945d4cfad48668268e66dd7f835e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:14.261975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-68b523f7-6d28-470e-b504-2c43a7c7ccb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:14.292033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 832/1636] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:14 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:14.315033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94926086-e3f4-4780-b06b-02f8efe09a98 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:14.344213 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:14.400391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 805/1637] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:14 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:14.400391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3065766c-e41e-414d-a477-513ce56293af None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:14.400391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3065766c-e41e-414d-a477-513ce56293af None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:14.400391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3065766c-e41e-414d-a477-513ce56293af None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:14.400391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 806/1638] 10.222.0.65 () {54 vars in 962 bytes} [Wed Aug 24 03:48:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:48:14.401457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94926086-e3f4-4780-b06b-02f8efe09a98 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:14.409782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec733ec-a6a5-437a-8730-d66f0c8d820d None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:14.409782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec733ec-a6a5-437a-8730-d66f0c8d820d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:14.409782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec733ec-a6a5-437a-8730-d66f0c8d820d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:14.476054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-8ec733ec-a6a5-437a-8730-d66f0c8d820d None None] MFA Rules not processed for user `e5896018712d4b82a781365e478ee855`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:14.478029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-8ec733ec-a6a5-437a-8730-d66f0c8d820d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:14.497749 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5e9a2c-1332-477a-9aca-5d9e9a70a644 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:14.498572 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5e9a2c-1332-477a-9aca-5d9e9a70a644 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:14.498858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5e9a2c-1332-477a-9aca-5d9e9a70a644 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:14.499133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5e9a2c-1332-477a-9aca-5d9e9a70a644 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:14.558875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:14.591458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 807/1639] 10.222.0.65 () {58 vars in 1059 bytes} [Wed Aug 24 03:48:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:14.592527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec733ec-a6a5-437a-8730-d66f0c8d820d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:14.593115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5e9a2c-1332-477a-9aca-5d9e9a70a644 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:14.595367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5e9a2c-1332-477a-9aca-5d9e9a70a644 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:14.608925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec733ec-a6a5-437a-8730-d66f0c8d820d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:14.609512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 833/1640] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:14 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:14.610679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5e9a2c-1332-477a-9aca-5d9e9a70a644 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:14.626199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5e9a2c-1332-477a-9aca-5d9e9a70a644 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:14.634135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde02391-9f4d-43e9-80a7-990644a0f3b7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:14.634939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde02391-9f4d-43e9-80a7-990644a0f3b7 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:14.635342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde02391-9f4d-43e9-80a7-990644a0f3b7 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:14.635751 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde02391-9f4d-43e9-80a7-990644a0f3b7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:14.636444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73a3637-2048-4022-b4f5-dbe417098fac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:14.637077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73a3637-2048-4022-b4f5-dbe417098fac service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:14.637354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73a3637-2048-4022-b4f5-dbe417098fac service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:14.639345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73a3637-2048-4022-b4f5-dbe417098fac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:14.684679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73a3637-2048-4022-b4f5-dbe417098fac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:14.688472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73a3637-2048-4022-b4f5-dbe417098fac service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:14.718893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 808/1641] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:14 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:14.756052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde02391-9f4d-43e9-80a7-990644a0f3b7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:14.758664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde02391-9f4d-43e9-80a7-990644a0f3b7 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:14.772342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 834/1642] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:14 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:15.341635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22eea8c-3706-487e-aab0-26673345d661 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:15.341635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22eea8c-3706-487e-aab0-26673345d661 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:15.341635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22eea8c-3706-487e-aab0-26673345d661 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:15.341635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 809/1643] 10.222.0.65 () {54 vars in 912 bytes} [Wed Aug 24 03:48:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:48:15.353984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a11d63e-dc32-412f-bac0-9c0307f6fbdc None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:15.353984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a11d63e-dc32-412f-bac0-9c0307f6fbdc None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:15.353984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a11d63e-dc32-412f-bac0-9c0307f6fbdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:15.353984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-8a11d63e-dc32-412f-bac0-9c0307f6fbdc None None] MFA Rules not processed for user `e5896018712d4b82a781365e478ee855`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:15.353984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8a11d63e-dc32-412f-bac0-9c0307f6fbdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:15.353984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 835/1644] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:48:15 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:15.359516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22eea8c-3706-487e-aab0-26673345d661 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:15.359516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22eea8c-3706-487e-aab0-26673345d661 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:15.359516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5b421c-76c8-463b-a0e1-db067a87825c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:15.359516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5b421c-76c8-463b-a0e1-db067a87825c service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:15.361594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5b421c-76c8-463b-a0e1-db067a87825c service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:15.361594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5b421c-76c8-463b-a0e1-db067a87825c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:15.361594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5b421c-76c8-463b-a0e1-db067a87825c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:15.361594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5b421c-76c8-463b-a0e1-db067a87825c service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:15.361594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 810/1645] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:16.689466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a11d63e-dc32-412f-bac0-9c0307f6fbdc None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:16.738103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:16.840560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a11d63e-dc32-412f-bac0-9c0307f6fbdc None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:16.851336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fda45a2-388e-4a10-b593-11d32a1fd6c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:16.861520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fda45a2-388e-4a10-b593-11d32a1fd6c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:16.861520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fda45a2-388e-4a10-b593-11d32a1fd6c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:16.861520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fda45a2-388e-4a10-b593-11d32a1fd6c9 admin admin] PATH_INFO: `/v3/users/67811e5632874935b3672587176d5d7d` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:16.862539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fda45a2-388e-4a10-b593-11d32a1fd6c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=67811e5632874935b3672587176d5d7d)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:16.863304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: WARNING py.warnings [None req-8fda45a2-388e-4a10-b593-11d32a1fd6c9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:48:16.863304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: warnings.warn(msg) Aug 24 03:48:16.863304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: Aug 24 03:48:16.864546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fda45a2-388e-4a10-b593-11d32a1fd6c9 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:17.158327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8fda45a2-388e-4a10-b593-11d32a1fd6c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '67811e5632874935b3672587176d5d7d'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:17.160844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8fda45a2-388e-4a10-b593-11d32a1fd6c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '67811e5632874935b3672587176d5d7d'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:17.190857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8fda45a2-388e-4a10-b593-11d32a1fd6c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '67811e5632874935b3672587176d5d7d'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:17.198712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8fda45a2-388e-4a10-b593-11d32a1fd6c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '67811e5632874935b3672587176d5d7d'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:17.254118 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ba2947-9495-4fa5-a2cd-e10946a61adc None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:17.255364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ba2947-9495-4fa5-a2cd-e10946a61adc None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:17.255906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ba2947-9495-4fa5-a2cd-e10946a61adc None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:17.259596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 811/1646] 10.222.0.65 () {54 vars in 962 bytes} [Wed Aug 24 03:48:17 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:48:17.267397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adca5fc-9ddd-4d2c-a977-ca31b27c931b None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:17.268135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adca5fc-9ddd-4d2c-a977-ca31b27c931b None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:17.268880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adca5fc-9ddd-4d2c-a977-ca31b27c931b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:17.294556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1adca5fc-9ddd-4d2c-a977-ca31b27c931b None None] MFA Rules not processed for user `e5896018712d4b82a781365e478ee855`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:17.296117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1adca5fc-9ddd-4d2c-a977-ca31b27c931b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:17.307412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 812/1647] 10.222.0.65 () {58 vars in 1059 bytes} [Wed Aug 24 03:48:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:48:17.316057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8fda45a2-388e-4a10-b593-11d32a1fd6c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '67811e5632874935b3672587176d5d7d'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:17.324364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbe1d6a-b255-48b9-89d7-364822567d8f None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:17.324658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbe1d6a-b255-48b9-89d7-364822567d8f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:17.324934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbe1d6a-b255-48b9-89d7-364822567d8f None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:17.329357 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 813/1648] 10.222.0.65 () {54 vars in 912 bytes} [Wed Aug 24 03:48:17 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:48:17.334578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386119c8-dd38-4334-a15b-364031232a74 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:17.334868 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386119c8-dd38-4334-a15b-364031232a74 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:17.335148 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386119c8-dd38-4334-a15b-364031232a74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:17.376207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 836/1649] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:48:16 2022] DELETE /identity/v3/users/67811e5632874935b3672587176d5d7d => generated 0 bytes in 685 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:17.377923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb04881-46b6-41c4-968f-c91f1852012b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:17.378257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb04881-46b6-41c4-968f-c91f1852012b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:17.378538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb04881-46b6-41c4-968f-c91f1852012b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:17.406645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-386119c8-dd38-4334-a15b-364031232a74 None None] MFA Rules not processed for user `e5896018712d4b82a781365e478ee855`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:17.407882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-386119c8-dd38-4334-a15b-364031232a74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:17.430442 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1cb04881-46b6-41c4-968f-c91f1852012b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:17.434776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1cb04881-46b6-41c4-968f-c91f1852012b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:17.558922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 814/1650] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:48:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:48:17.574107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386119c8-dd38-4334-a15b-364031232a74 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:17.576399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-386119c8-dd38-4334-a15b-364031232a74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:17.588342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 837/1651] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:17.630428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb04881-46b6-41c4-968f-c91f1852012b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:17.632415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1cb04881-46b6-41c4-968f-c91f1852012b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:17.663742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386119c8-dd38-4334-a15b-364031232a74 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:17.668942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc57b035-5631-495f-8868-63b742065ef9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:17.669639 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc57b035-5631-495f-8868-63b742065ef9 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:17.669923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc57b035-5631-495f-8868-63b742065ef9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:17.670301 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc57b035-5631-495f-8868-63b742065ef9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:17.683255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc57b035-5631-495f-8868-63b742065ef9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:17.685323 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc57b035-5631-495f-8868-63b742065ef9 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:17.704348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 815/1652] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:17.799502 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:17.816349 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb04881-46b6-41c4-968f-c91f1852012b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:17.831097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd253dd2-414b-4e30-be84-75e5ff1e7e65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:17.832186 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd253dd2-414b-4e30-be84-75e5ff1e7e65 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:17.832693 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd253dd2-414b-4e30-be84-75e5ff1e7e65 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:17.833061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd253dd2-414b-4e30-be84-75e5ff1e7e65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:17.871762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd253dd2-414b-4e30-be84-75e5ff1e7e65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:17.873989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd253dd2-414b-4e30-be84-75e5ff1e7e65 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:17.891443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 838/1653] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:48:19.989573 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc57b035-5631-495f-8868-63b742065ef9 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:19.990073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-bc57b035-5631-495f-8868-63b742065ef9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:20.020927 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc57b035-5631-495f-8868-63b742065ef9 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:20.038546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:20.039550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:20.039731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:20.039977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] PATH_INFO: `/v3/projects/a180e79a3883412b8d1e24b48a7b69a4` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:20.042933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] RBAC: Authorizing `identity:delete_project(project_id=a180e79a3883412b8d1e24b48a7b69a4)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:20.043977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: WARNING py.warnings [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 24 03:48:20.043977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: warnings.warn(msg) Aug 24 03:48:20.043977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: Aug 24 03:48:20.047114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:20.247649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:20.263120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a180e79a3883412b8d1e24b48a7b69a4', 'request_id': 'req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:20.274333 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a180e79a3883412b8d1e24b48a7b69a4', 'request_id': 'req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:20.314346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a180e79a3883412b8d1e24b48a7b69a4', 'request_id': 'req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:20.440499 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] The token cache is being invalidate because project a180e79a3883412b8d1e24b48a7b69a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:48:20.440852 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:20.441538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:20.443365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 816/1654] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:48:19 2022] DELETE /identity/v3/projects/a180e79a3883412b8d1e24b48a7b69a4 => generated 0 bytes in 456 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:20.448196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd253dd2-414b-4e30-be84-75e5ff1e7e65 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:20.462820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd253dd2-414b-4e30-be84-75e5ff1e7e65 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:20.598983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a2b91-ba29-4d75-bd14-2b627b47b6bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:20.599909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a2b91-ba29-4d75-bd14-2b627b47b6bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:20.600348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a2b91-ba29-4d75-bd14-2b627b47b6bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:20.600803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a2b91-ba29-4d75-bd14-2b627b47b6bc admin admin] PATH_INFO: `/v3/users/462945d4cfad48668268e66dd7f835e5` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:20.603642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a2b91-ba29-4d75-bd14-2b627b47b6bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=462945d4cfad48668268e66dd7f835e5)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:20.606719 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a2b91-ba29-4d75-bd14-2b627b47b6bc admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:20.994007 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-195a2b91-ba29-4d75-bd14-2b627b47b6bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '462945d4cfad48668268e66dd7f835e5'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:20.995412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-195a2b91-ba29-4d75-bd14-2b627b47b6bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '462945d4cfad48668268e66dd7f835e5'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:21.014335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-195a2b91-ba29-4d75-bd14-2b627b47b6bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '462945d4cfad48668268e66dd7f835e5'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:21.015804 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-195a2b91-ba29-4d75-bd14-2b627b47b6bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '462945d4cfad48668268e66dd7f835e5'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:21.126620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-195a2b91-ba29-4d75-bd14-2b627b47b6bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '462945d4cfad48668268e66dd7f835e5'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:21.191325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 839/1655] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:48:20 2022] DELETE /identity/v3/users/462945d4cfad48668268e66dd7f835e5 => generated 0 bytes in 741 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:23.114892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:23.117490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:23.157959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16bb2aa-038f-44a7-b0d7-aaed0671aecb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:23.345936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f2a979-251e-41bb-a39b-c85a3b974757 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:23.347027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f2a979-251e-41bb-a39b-c85a3b974757 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:23.347027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f2a979-251e-41bb-a39b-c85a3b974757 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:23.347311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f2a979-251e-41bb-a39b-c85a3b974757 admin admin] PATH_INFO: `/v3/projects/0030b901e26545a5a136cbcbbab21b5e` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:23.348586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f2a979-251e-41bb-a39b-c85a3b974757 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0030b901e26545a5a136cbcbbab21b5e)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:23.350518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f2a979-251e-41bb-a39b-c85a3b974757 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:23.589845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-07f2a979-251e-41bb-a39b-c85a3b974757 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0030b901e26545a5a136cbcbbab21b5e', 'request_id': 'req-07f2a979-251e-41bb-a39b-c85a3b974757'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:23.607504 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-07f2a979-251e-41bb-a39b-c85a3b974757 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0030b901e26545a5a136cbcbbab21b5e', 'request_id': 'req-07f2a979-251e-41bb-a39b-c85a3b974757'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:23.634743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-07f2a979-251e-41bb-a39b-c85a3b974757 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0030b901e26545a5a136cbcbbab21b5e', 'request_id': 'req-07f2a979-251e-41bb-a39b-c85a3b974757'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:23.794287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-07f2a979-251e-41bb-a39b-c85a3b974757 admin admin] The token cache is being invalidate because project 0030b901e26545a5a136cbcbbab21b5e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:48:23.796858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-07f2a979-251e-41bb-a39b-c85a3b974757 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:23.799375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-07f2a979-251e-41bb-a39b-c85a3b974757 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:23.809438 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 817/1656] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:48:23 2022] DELETE /identity/v3/projects/0030b901e26545a5a136cbcbbab21b5e => generated 0 bytes in 695 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:23.814655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313ca27c-4c17-4ebc-9965-2bc5bcf2e0bf None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:23.815543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313ca27c-4c17-4ebc-9965-2bc5bcf2e0bf None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:23.816227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313ca27c-4c17-4ebc-9965-2bc5bcf2e0bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:23.869582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-313ca27c-4c17-4ebc-9965-2bc5bcf2e0bf None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:23.871473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-313ca27c-4c17-4ebc-9965-2bc5bcf2e0bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:24.025919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d657e10-fcf3-41f2-93bd-5fb88323534b None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:24.026479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d657e10-fcf3-41f2-93bd-5fb88323534b None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:24.026901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d657e10-fcf3-41f2-93bd-5fb88323534b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:24.035324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 840/1657] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:24.053797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-3d657e10-fcf3-41f2-93bd-5fb88323534b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:24.056983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3d657e10-fcf3-41f2-93bd-5fb88323534b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:24.085838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 818/1658] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:24.087483 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313ca27c-4c17-4ebc-9965-2bc5bcf2e0bf None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:24.092928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:24.102180 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313ca27c-4c17-4ebc-9965-2bc5bcf2e0bf None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:24.114131 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa53166-a413-4e77-9513-bd67226a1ca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:24.115046 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa53166-a413-4e77-9513-bd67226a1ca2 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:24.115621 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa53166-a413-4e77-9513-bd67226a1ca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:24.116035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa53166-a413-4e77-9513-bd67226a1ca2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:24.118531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa53166-a413-4e77-9513-bd67226a1ca2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:24.128511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa53166-a413-4e77-9513-bd67226a1ca2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:24.130413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa53166-a413-4e77-9513-bd67226a1ca2 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:24.150884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d657e10-fcf3-41f2-93bd-5fb88323534b None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:24.161936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 841/1659] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:48:24.166036 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d657e10-fcf3-41f2-93bd-5fb88323534b None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:24.168670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a162be13-5668-4962-b11c-c155fb4c3124 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:24.169491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a162be13-5668-4962-b11c-c155fb4c3124 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:24.169909 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a162be13-5668-4962-b11c-c155fb4c3124 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:24.170373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a162be13-5668-4962-b11c-c155fb4c3124 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:24.171251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a162be13-5668-4962-b11c-c155fb4c3124 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:24.173530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a162be13-5668-4962-b11c-c155fb4c3124 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:24.341913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 819/1660] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:24 2022] POST /identity/v3/projects => generated 379 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:48:24.346536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa53166-a413-4e77-9513-bd67226a1ca2 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:24.370792 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa53166-a413-4e77-9513-bd67226a1ca2 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:24.548234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4de1721-8831-44f3-8d25-95baf7075deb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:24.549156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4de1721-8831-44f3-8d25-95baf7075deb admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:24.549579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4de1721-8831-44f3-8d25-95baf7075deb admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:24.549989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4de1721-8831-44f3-8d25-95baf7075deb admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:24.551287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4de1721-8831-44f3-8d25-95baf7075deb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:24.564467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4de1721-8831-44f3-8d25-95baf7075deb admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:24.808782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 842/1661] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:24 2022] POST /identity/v3/users => generated 290 bytes in 463 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:48:24.814334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a162be13-5668-4962-b11c-c155fb4c3124 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:24.832143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a162be13-5668-4962-b11c-c155fb4c3124 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:24.845219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c19c8bc-9add-4051-b29b-536931bbd7f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:24.846124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c19c8bc-9add-4051-b29b-536931bbd7f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:24.846581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c19c8bc-9add-4051-b29b-536931bbd7f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:24.846986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c19c8bc-9add-4051-b29b-536931bbd7f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:24.847793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c19c8bc-9add-4051-b29b-536931bbd7f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:24.849630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c19c8bc-9add-4051-b29b-536931bbd7f6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:24.885368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 820/1662] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:24 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:24.886376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4de1721-8831-44f3-8d25-95baf7075deb admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:24.897034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4de1721-8831-44f3-8d25-95baf7075deb admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:24.907454 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042dc7a9-efbb-461c-b815-7e142b5674e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:24.908101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042dc7a9-efbb-461c-b815-7e142b5674e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:24.908381 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042dc7a9-efbb-461c-b815-7e142b5674e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:24.908662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042dc7a9-efbb-461c-b815-7e142b5674e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:24.909312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042dc7a9-efbb-461c-b815-7e142b5674e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:24.911383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042dc7a9-efbb-461c-b815-7e142b5674e4 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:24.949440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 843/1663] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:24 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:24.953793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c19c8bc-9add-4051-b29b-536931bbd7f6 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:24.970439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c19c8bc-9add-4051-b29b-536931bbd7f6 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:24.987327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc23a6d7-ca7b-451b-ab81-9412329298eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:24.989539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc23a6d7-ca7b-451b-ab81-9412329298eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:24.990348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc23a6d7-ca7b-451b-ab81-9412329298eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:24.992332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc23a6d7-ca7b-451b-ab81-9412329298eb admin admin] PATH_INFO: `/v3/projects/47ed67b528624b85896220c806722c6a/users/5abc95284c82495db9b04188411c3d90/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:25.049259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc23a6d7-ca7b-451b-ab81-9412329298eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=47ed67b528624b85896220c806722c6a, user_id=5abc95284c82495db9b04188411c3d90, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:25.056484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc23a6d7-ca7b-451b-ab81-9412329298eb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:25.137952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:25.145011 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 821/1664] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:24 2022] PUT /identity/v3/projects/47ed67b528624b85896220c806722c6a/users/5abc95284c82495db9b04188411c3d90/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:25.155813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490ebc77-1213-48e8-bbc2-7e0c14ee8652 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:25.157281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490ebc77-1213-48e8-bbc2-7e0c14ee8652 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:25.157626 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490ebc77-1213-48e8-bbc2-7e0c14ee8652 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:25.220803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-490ebc77-1213-48e8-bbc2-7e0c14ee8652 None None] MFA Rules not processed for user `5abc95284c82495db9b04188411c3d90`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:25.226751 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-490ebc77-1213-48e8-bbc2-7e0c14ee8652 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:25.444069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 844/1665] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:25 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:25.446675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc23a6d7-ca7b-451b-ab81-9412329298eb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:25.594700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc23a6d7-ca7b-451b-ab81-9412329298eb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:25.602075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33ff332-8d3a-4ec3-ba34-bfda9587afb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:25.603255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33ff332-8d3a-4ec3-ba34-bfda9587afb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:25.604378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33ff332-8d3a-4ec3-ba34-bfda9587afb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:25.604819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33ff332-8d3a-4ec3-ba34-bfda9587afb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:25.605691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33ff332-8d3a-4ec3-ba34-bfda9587afb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:25.608515 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33ff332-8d3a-4ec3-ba34-bfda9587afb2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:25.787579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 822/1666] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:25 2022] POST /identity/v3/projects => generated 381 bytes in 343 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:25.792405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490ebc77-1213-48e8-bbc2-7e0c14ee8652 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:25.818799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490ebc77-1213-48e8-bbc2-7e0c14ee8652 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:26.012185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:26.037547 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71670a94-ee72-40e5-baf8-8a1278f879ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:26.038385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71670a94-ee72-40e5-baf8-8a1278f879ce admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:26.039145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71670a94-ee72-40e5-baf8-8a1278f879ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:26.039430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71670a94-ee72-40e5-baf8-8a1278f879ce admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:26.040089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71670a94-ee72-40e5-baf8-8a1278f879ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:26.049310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71670a94-ee72-40e5-baf8-8a1278f879ce admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:26.185017 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 845/1667] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:25 2022] POST /identity/v3/users => generated 297 bytes in 393 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:26.191681 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33ff332-8d3a-4ec3-ba34-bfda9587afb2 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:26.210372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33ff332-8d3a-4ec3-ba34-bfda9587afb2 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:26.228418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1d7ab5-92d3-4577-8d93-a4bde12bd1e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:26.229072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1d7ab5-92d3-4577-8d93-a4bde12bd1e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:26.229356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1d7ab5-92d3-4577-8d93-a4bde12bd1e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:26.235731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1d7ab5-92d3-4577-8d93-a4bde12bd1e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:26.236522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1d7ab5-92d3-4577-8d93-a4bde12bd1e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:26.238185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1d7ab5-92d3-4577-8d93-a4bde12bd1e9 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:26.285731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71670a94-ee72-40e5-baf8-8a1278f879ce admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:26.296016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 823/1668] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:26 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:26.299956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71670a94-ee72-40e5-baf8-8a1278f879ce admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:26.309319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faad0178-8aa2-43bd-9909-4da3e259b8b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:26.310204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faad0178-8aa2-43bd-9909-4da3e259b8b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:26.311699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faad0178-8aa2-43bd-9909-4da3e259b8b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:26.312086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faad0178-8aa2-43bd-9909-4da3e259b8b2 admin admin] PATH_INFO: `/v3/projects/43930501ae7e4d2ea740c83c17880de9/users/36ac3171e69e4b16946fb1a3d25b1a2b/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:26.337431 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faad0178-8aa2-43bd-9909-4da3e259b8b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43930501ae7e4d2ea740c83c17880de9, user_id=36ac3171e69e4b16946fb1a3d25b1a2b, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:26.341357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faad0178-8aa2-43bd-9909-4da3e259b8b2 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:26.466301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 846/1669] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:26 2022] PUT /identity/v3/projects/43930501ae7e4d2ea740c83c17880de9/users/36ac3171e69e4b16946fb1a3d25b1a2b/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:26.469508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1d7ab5-92d3-4577-8d93-a4bde12bd1e9 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:26.485829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1d7ab5-92d3-4577-8d93-a4bde12bd1e9 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:26.635126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296f8c53-3856-4dc3-8964-c96bb72192b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:26.636051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296f8c53-3856-4dc3-8964-c96bb72192b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:26.636476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296f8c53-3856-4dc3-8964-c96bb72192b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:26.636891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296f8c53-3856-4dc3-8964-c96bb72192b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:26.637691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296f8c53-3856-4dc3-8964-c96bb72192b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:26.642475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296f8c53-3856-4dc3-8964-c96bb72192b8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:26.659052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 824/1670] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:26 2022] GET /identity/v3/roles => generated 1420 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:26.664244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faad0178-8aa2-43bd-9909-4da3e259b8b2 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:26.695418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faad0178-8aa2-43bd-9909-4da3e259b8b2 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:26.727270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d095f-16aa-4a67-9e66-05896fcb6025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:26.731008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d095f-16aa-4a67-9e66-05896fcb6025 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:26.731317 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d095f-16aa-4a67-9e66-05896fcb6025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:26.731603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d095f-16aa-4a67-9e66-05896fcb6025 admin admin] PATH_INFO: `/v3/projects/43930501ae7e4d2ea740c83c17880de9/users/36ac3171e69e4b16946fb1a3d25b1a2b/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:26.738266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00d095f-16aa-4a67-9e66-05896fcb6025 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43930501ae7e4d2ea740c83c17880de9, user_id=36ac3171e69e4b16946fb1a3d25b1a2b, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:26.745503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00d095f-16aa-4a67-9e66-05896fcb6025 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:26.835592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3728c60-48e6-4b65-b9fb-0461d92e8b35 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:26.836246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3728c60-48e6-4b65-b9fb-0461d92e8b35 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:26.836519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 847/1671] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:26 2022] PUT /identity/v3/projects/43930501ae7e4d2ea740c83c17880de9/users/36ac3171e69e4b16946fb1a3d25b1a2b/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:26.836844 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3728c60-48e6-4b65-b9fb-0461d92e8b35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:26.909221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-c3728c60-48e6-4b65-b9fb-0461d92e8b35 None None] MFA Rules not processed for user `36ac3171e69e4b16946fb1a3d25b1a2b`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:26.910686 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c3728c60-48e6-4b65-b9fb-0461d92e8b35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:27.078216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:27.106606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 825/1672] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:26 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:27.137862 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d095f-16aa-4a67-9e66-05896fcb6025 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:27.141930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b00d095f-16aa-4a67-9e66-05896fcb6025 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:27.296942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d095f-16aa-4a67-9e66-05896fcb6025 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:27.306518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ceb585-7912-4892-8305-efcef6581898 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:27.307916 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ceb585-7912-4892-8305-efcef6581898 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:27.309401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ceb585-7912-4892-8305-efcef6581898 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:27.312029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ceb585-7912-4892-8305-efcef6581898 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:27.338530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ceb585-7912-4892-8305-efcef6581898 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:27.340033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ceb585-7912-4892-8305-efcef6581898 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:27.359906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 848/1673] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:27 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:28.967869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3728c60-48e6-4b65-b9fb-0461d92e8b35 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:28.982223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3728c60-48e6-4b65-b9fb-0461d92e8b35 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:28.996314 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3fecda-edb9-436d-acb3-158dfca9c8bd tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ac3171e69e4b16946fb1a3d25b1a2b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '43930501ae7e4d2ea740c83c17880de9', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:28.997137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3fecda-edb9-436d-acb3-158dfca9c8bd tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:28.997575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3fecda-edb9-436d-acb3-158dfca9c8bd tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:28.998000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3fecda-edb9-436d-acb3-158dfca9c8bd tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:28.999022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3fecda-edb9-436d-acb3-158dfca9c8bd tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:28.999489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3fecda-edb9-436d-acb3-158dfca9c8bd tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:29.001547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3fecda-edb9-436d-acb3-158dfca9c8bd tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:29.027824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 826/1674] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:48:29.032116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ceb585-7912-4892-8305-efcef6581898 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:29.048266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ceb585-7912-4892-8305-efcef6581898 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:29.056090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40a5a49-5833-4cdf-a625-bf347b5c6291 tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ac3171e69e4b16946fb1a3d25b1a2b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '43930501ae7e4d2ea740c83c17880de9', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:29.057153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40a5a49-5833-4cdf-a625-bf347b5c6291 tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:29.057761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40a5a49-5833-4cdf-a625-bf347b5c6291 tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:29.058330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40a5a49-5833-4cdf-a625-bf347b5c6291 tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:29.059147 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40a5a49-5833-4cdf-a625-bf347b5c6291 tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:29.061817 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40a5a49-5833-4cdf-a625-bf347b5c6291 tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:29.248265 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 849/1675] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:29 2022] POST /identity/v3/projects => generated 355 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:48:29.251640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3fecda-edb9-436d-acb3-158dfca9c8bd tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:29.268633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3fecda-edb9-436d-acb3-158dfca9c8bd tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:29.417101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca37e23-0b58-476d-b8bb-06d341eb01ec None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:29.418520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca37e23-0b58-476d-b8bb-06d341eb01ec None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:29.420864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca37e23-0b58-476d-b8bb-06d341eb01ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:29.438619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c9e165-1e3b-47bd-948b-219cfc156ebc tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ac3171e69e4b16946fb1a3d25b1a2b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '43930501ae7e4d2ea740c83c17880de9', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:29.442682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c9e165-1e3b-47bd-948b-219cfc156ebc tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:29.445876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c9e165-1e3b-47bd-948b-219cfc156ebc tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:29.447386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c9e165-1e3b-47bd-948b-219cfc156ebc tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:29.452952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c9e165-1e3b-47bd-948b-219cfc156ebc tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:29.455521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c9e165-1e3b-47bd-948b-219cfc156ebc tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:29.463456 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c9e165-1e3b-47bd-948b-219cfc156ebc tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:29.525493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 827/1676] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:48:29.534603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-7ca37e23-0b58-476d-b8bb-06d341eb01ec None None] MFA Rules not processed for user `0772f8b4306f4e239f075f7b1c13790f`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:29.539203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7ca37e23-0b58-476d-b8bb-06d341eb01ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:29.633740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c9e165-1e3b-47bd-948b-219cfc156ebc tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:29.649915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c9e165-1e3b-47bd-948b-219cfc156ebc tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:29.671860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ac3171e69e4b16946fb1a3d25b1a2b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '43930501ae7e4d2ea740c83c17880de9', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:29.672821 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:29.673288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:29.673871 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] PATH_INFO: `/v3/projects/68dd74f749a8498aa5312c3e8e01f3b7` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:29.675915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: Authorizing `identity:delete_project(project_id=68dd74f749a8498aa5312c3e8e01f3b7)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:29.692843 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:29.765116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 850/1677] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:29 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 354 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:29.767847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fcfe2b-788c-4ac1-b7a5-eb01d6516ffe None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:29.768255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fcfe2b-788c-4ac1-b7a5-eb01d6516ffe None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:29.769305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fcfe2b-788c-4ac1-b7a5-eb01d6516ffe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:29.847897 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-06fcfe2b-788c-4ac1-b7a5-eb01d6516ffe None None] MFA Rules not processed for user `77b4b56f008c44c081130377cf8ce630`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:29.857923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-06fcfe2b-788c-4ac1-b7a5-eb01d6516ffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:29.879952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:29.920770 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '68dd74f749a8498aa5312c3e8e01f3b7', 'request_id': 'req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:29.948220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '68dd74f749a8498aa5312c3e8e01f3b7', 'request_id': 'req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:29.979463 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '68dd74f749a8498aa5312c3e8e01f3b7', 'request_id': 'req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:30.046719 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 851/1678] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:29 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:30.048427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9d416f-0f9c-4a11-acf8-7500ce90d02d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:30.048427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9d416f-0f9c-4a11-acf8-7500ce90d02d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:30.049392 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9d416f-0f9c-4a11-acf8-7500ce90d02d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:30.099680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] The token cache is being invalidate because project 68dd74f749a8498aa5312c3e8e01f3b7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:48:30.105421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:30.105421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:30.105830 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-ea9d416f-0f9c-4a11-acf8-7500ce90d02d None None] MFA Rules not processed for user `0772f8b4306f4e239f075f7b1c13790f`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:30.107841 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ea9d416f-0f9c-4a11-acf8-7500ce90d02d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:30.108616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 828/1679] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:48:29 2022] DELETE /identity/v3/projects/68dd74f749a8498aa5312c3e8e01f3b7 => generated 0 bytes in 472 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:30.109619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:30.122216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:30.288992 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 852/1680] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:30 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:30.298193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:30.320900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a324d31a-bc54-412a-acd3-4fb4df8b2e0b tempest-VolumeQuotaClassesTest-1671317329 tempest-VolumeQuotaClassesTest-1671317329-project-admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:30.322871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9d416f-0f9c-4a11-acf8-7500ce90d02d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:30.325524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ea9d416f-0f9c-4a11-acf8-7500ce90d02d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:30.327626 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e77e6d-76fa-4503-a9c4-0010e973b3ee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:30.328515 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e77e6d-76fa-4503-a9c4-0010e973b3ee service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:30.328799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e77e6d-76fa-4503-a9c4-0010e973b3ee service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:30.329075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e77e6d-76fa-4503-a9c4-0010e973b3ee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:30.331160 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b7e77e6d-76fa-4503-a9c4-0010e973b3ee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:30.359691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e77e6d-76fa-4503-a9c4-0010e973b3ee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:30.361546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e77e6d-76fa-4503-a9c4-0010e973b3ee service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:30.386391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9d416f-0f9c-4a11-acf8-7500ce90d02d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:30.388704 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 829/1681] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:30 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:30.429547 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2345e99-2cea-4c57-87df-cb21f2a79abf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:30.430584 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2345e99-2cea-4c57-87df-cb21f2a79abf service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:30.430859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2345e99-2cea-4c57-87df-cb21f2a79abf service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:30.431294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2345e99-2cea-4c57-87df-cb21f2a79abf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:30.447319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d2345e99-2cea-4c57-87df-cb21f2a79abf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:30.478718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2345e99-2cea-4c57-87df-cb21f2a79abf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:30.506036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2345e99-2cea-4c57-87df-cb21f2a79abf service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:30.557020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020f9359-8362-4021-8608-ee4bb2e5e003 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:30.559258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020f9359-8362-4021-8608-ee4bb2e5e003 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:30.559931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020f9359-8362-4021-8608-ee4bb2e5e003 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:30.568836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 853/1682] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:30 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 246 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:30.620409 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-020f9359-8362-4021-8608-ee4bb2e5e003 None None] MFA Rules not processed for user `bd8d967d9ecb410ea47e130ac3c8b4a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:30.622700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-020f9359-8362-4021-8608-ee4bb2e5e003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:30.765354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 830/1683] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:30 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:48:31.174139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2345e99-2cea-4c57-87df-cb21f2a79abf service nova] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:31.201889 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2345e99-2cea-4c57-87df-cb21f2a79abf service nova] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:31.218821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26ba21-3f52-4346-a24f-b7865385f16b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:31.219463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26ba21-3f52-4346-a24f-b7865385f16b service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:31.223766 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26ba21-3f52-4346-a24f-b7865385f16b service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:31.224180 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26ba21-3f52-4346-a24f-b7865385f16b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:31.253836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f26ba21-3f52-4346-a24f-b7865385f16b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:31.257164 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f26ba21-3f52-4346-a24f-b7865385f16b service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:31.294921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 854/1684] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:31 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:31.564558 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf05453-b958-4b75-9545-575b4196e440 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:31.565080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf05453-b958-4b75-9545-575b4196e440 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:31.565080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf05453-b958-4b75-9545-575b4196e440 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:31.566699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 831/1685] 10.222.0.65 () {54 vars in 912 bytes} [Wed Aug 24 03:48:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:48:31.580539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db8d200-7de1-4149-9743-e76e58c951e7 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:31.583678 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db8d200-7de1-4149-9743-e76e58c951e7 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:31.584212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db8d200-7de1-4149-9743-e76e58c951e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:31.655838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf05453-b958-4b75-9545-575b4196e440 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:31.658046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7cf05453-b958-4b75-9545-575b4196e440 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:31.701962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-0db8d200-7de1-4149-9743-e76e58c951e7 None None] MFA Rules not processed for user `64f23f0c8abc406b9700361723623387`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:31.714699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0db8d200-7de1-4149-9743-e76e58c951e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:31.824420 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 855/1686] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:48:31 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:31.825397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf05453-b958-4b75-9545-575b4196e440 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:31.827011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db8d200-7de1-4149-9743-e76e58c951e7 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:31.829784 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0db8d200-7de1-4149-9743-e76e58c951e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:31.841938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:31.841938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:31.841938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:31.841938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] PATH_INFO: `/v3/users/5abc95284c82495db9b04188411c3d90` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:31.842637 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5abc95284c82495db9b04188411c3d90)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:31.848188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:31.862500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db8d200-7de1-4149-9743-e76e58c951e7 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:31.877225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6792aeb-731f-41b8-acd1-a4bd0dc70b23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:31.877894 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6792aeb-731f-41b8-acd1-a4bd0dc70b23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:31.878219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6792aeb-731f-41b8-acd1-a4bd0dc70b23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:31.878504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6792aeb-731f-41b8-acd1-a4bd0dc70b23 admin admin] PATH_INFO: `/v3/users/28150f2ee18e47c783f906a162e568bb` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:31.880660 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6792aeb-731f-41b8-acd1-a4bd0dc70b23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=28150f2ee18e47c783f906a162e568bb)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:31.895021 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6792aeb-731f-41b8-acd1-a4bd0dc70b23 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:31.969538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:32.187530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5abc95284c82495db9b04188411c3d90'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:32.190888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5abc95284c82495db9b04188411c3d90'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:32.220936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5abc95284c82495db9b04188411c3d90'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:32.250543 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5abc95284c82495db9b04188411c3d90'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:32.253970 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f6792aeb-731f-41b8-acd1-a4bd0dc70b23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28150f2ee18e47c783f906a162e568bb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:32.263632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f6792aeb-731f-41b8-acd1-a4bd0dc70b23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28150f2ee18e47c783f906a162e568bb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:32.281953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f6792aeb-731f-41b8-acd1-a4bd0dc70b23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28150f2ee18e47c783f906a162e568bb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:32.286085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f6792aeb-731f-41b8-acd1-a4bd0dc70b23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28150f2ee18e47c783f906a162e568bb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:32.418765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5abc95284c82495db9b04188411c3d90'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:32.427261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 832/1687] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:48:31 2022] DELETE /identity/v3/users/5abc95284c82495db9b04188411c3d90 => generated 0 bytes in 772 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:32.431059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:32.434635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:32.543969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f6792aeb-731f-41b8-acd1-a4bd0dc70b23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28150f2ee18e47c783f906a162e568bb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:32.598101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 856/1688] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:48:31 2022] DELETE /identity/v3/users/28150f2ee18e47c783f906a162e568bb => generated 0 bytes in 773 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:32.618292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c1719e-7cc3-4df3-bb25-a5d020155569 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:32.619027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c1719e-7cc3-4df3-bb25-a5d020155569 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:32.620037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c1719e-7cc3-4df3-bb25-a5d020155569 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:32.669853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ff4c05-5e39-45a1-9f1e-3111d0f70824 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:32.724885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e6c1719e-7cc3-4df3-bb25-a5d020155569 None None] MFA Rules not processed for user `7f4991f9c6ca4fc99f756f75f9460808`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:32.724885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e6c1719e-7cc3-4df3-bb25-a5d020155569 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:32.936733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e837bf-456f-4013-81eb-0ec47a7b737c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:32.936733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e837bf-456f-4013-81eb-0ec47a7b737c service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:32.936733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e837bf-456f-4013-81eb-0ec47a7b737c service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:32.936733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e837bf-456f-4013-81eb-0ec47a7b737c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:32.938124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f1e837bf-456f-4013-81eb-0ec47a7b737c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:32.938124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e837bf-456f-4013-81eb-0ec47a7b737c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:32.938124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e837bf-456f-4013-81eb-0ec47a7b737c service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:32.938632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 857/1689] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:32 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:32.938632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4cd923-3be2-4985-af4a-6113d2b4cfbc None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:32.938632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4cd923-3be2-4985-af4a-6113d2b4cfbc None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:32.938632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4cd923-3be2-4985-af4a-6113d2b4cfbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:32.938632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-6b4cd923-3be2-4985-af4a-6113d2b4cfbc None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:32.939005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 833/1690] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:32 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 507 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:32.939096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6b4cd923-3be2-4985-af4a-6113d2b4cfbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:32.939181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7699545-5b8a-4882-ba6c-85e200aa48e9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:32.939181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7699545-5b8a-4882-ba6c-85e200aa48e9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:32.943232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7699545-5b8a-4882-ba6c-85e200aa48e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:33.009402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f7699545-5b8a-4882-ba6c-85e200aa48e9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:33.018350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f7699545-5b8a-4882-ba6c-85e200aa48e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:33.050026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 858/1691] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:33.050977 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4cd923-3be2-4985-af4a-6113d2b4cfbc None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:33.053009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6b4cd923-3be2-4985-af4a-6113d2b4cfbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:33.196513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 834/1692] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:33.197378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7699545-5b8a-4882-ba6c-85e200aa48e9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:33.200740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f7699545-5b8a-4882-ba6c-85e200aa48e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:33.224055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4cd923-3be2-4985-af4a-6113d2b4cfbc None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:33.232086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aff82e3-eab6-47fa-b957-e9059fe29e22 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:33.233206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aff82e3-eab6-47fa-b957-e9059fe29e22 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:33.233795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aff82e3-eab6-47fa-b957-e9059fe29e22 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:33.236022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aff82e3-eab6-47fa-b957-e9059fe29e22 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:33.251973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aff82e3-eab6-47fa-b957-e9059fe29e22 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:33.254625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aff82e3-eab6-47fa-b957-e9059fe29e22 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:33.261706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7699545-5b8a-4882-ba6c-85e200aa48e9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:33.271084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 859/1693] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:33 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:33.274305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc327a1-c122-4c4a-bdcd-ea297c3ba1ee None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:33.274775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc327a1-c122-4c4a-bdcd-ea297c3ba1ee None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:33.275200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc327a1-c122-4c4a-bdcd-ea297c3ba1ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:33.285347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7ede71-c0db-4194-9f49-59a09a7423cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:33.285448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ede71-c0db-4194-9f49-59a09a7423cf service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:33.285724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ede71-c0db-4194-9f49-59a09a7423cf service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:33.286030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ede71-c0db-4194-9f49-59a09a7423cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:33.344054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7ede71-c0db-4194-9f49-59a09a7423cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:33.345545 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7ede71-c0db-4194-9f49-59a09a7423cf service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:33.391538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-6dc327a1-c122-4c4a-bdcd-ea297c3ba1ee None None] MFA Rules not processed for user `728c3f2735d6441b870cf146fbf084c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:33.397379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6dc327a1-c122-4c4a-bdcd-ea297c3ba1ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:33.402208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:33.404592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 835/1694] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:33.405939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7ede71-c0db-4194-9f49-59a09a7423cf service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:33.460076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7ede71-c0db-4194-9f49-59a09a7423cf service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:33.480633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49170c7c-497e-4bf5-8187-55dca85de333 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:33.480633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49170c7c-497e-4bf5-8187-55dca85de333 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:33.480633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49170c7c-497e-4bf5-8187-55dca85de333 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:33.480633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49170c7c-497e-4bf5-8187-55dca85de333 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:33.492905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49170c7c-497e-4bf5-8187-55dca85de333 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:33.494748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49170c7c-497e-4bf5-8187-55dca85de333 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:33.526822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 836/1695] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:33.598284 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 860/1696] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:33 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 326 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:33.609938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a0166f-d77a-43dc-a51c-3f44414275fb None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:33.610559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a0166f-d77a-43dc-a51c-3f44414275fb None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:33.610559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a0166f-d77a-43dc-a51c-3f44414275fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:33.678136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-59a0166f-d77a-43dc-a51c-3f44414275fb None None] MFA Rules not processed for user `728c3f2735d6441b870cf146fbf084c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:33.680818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-59a0166f-d77a-43dc-a51c-3f44414275fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:33.715166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 837/1697] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:33 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:33.728153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc327a1-c122-4c4a-bdcd-ea297c3ba1ee None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:33.755937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc327a1-c122-4c4a-bdcd-ea297c3ba1ee None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:33.799462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b883a02-222c-4c58-9813-5de4beea2801 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:33.800254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b883a02-222c-4c58-9813-5de4beea2801 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:33.804592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b883a02-222c-4c58-9813-5de4beea2801 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:33.804592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b883a02-222c-4c58-9813-5de4beea2801 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:33.840914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b883a02-222c-4c58-9813-5de4beea2801 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:33.850427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b883a02-222c-4c58-9813-5de4beea2801 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:33.885929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 861/1698] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:33 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:34.246606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:34.534332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6608ee7f-7e26-4bc3-9a0f-47ccb5323f78 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:34.534602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6608ee7f-7e26-4bc3-9a0f-47ccb5323f78 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:34.535027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6608ee7f-7e26-4bc3-9a0f-47ccb5323f78 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:34.536763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 838/1699] 10.222.0.65 () {54 vars in 961 bytes} [Wed Aug 24 03:48:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:48:34.551570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b031657f-5407-4603-8d0a-b3a6a9b81a6b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:34.551570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b031657f-5407-4603-8d0a-b3a6a9b81a6b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:34.551570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b031657f-5407-4603-8d0a-b3a6a9b81a6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:34.604429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-b031657f-5407-4603-8d0a-b3a6a9b81a6b None None] MFA Rules not processed for user `4b347c9c89a64168b51c439b4a92d647`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:34.605690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b031657f-5407-4603-8d0a-b3a6a9b81a6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:34.710658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece01850-0276-4633-8ccd-7004b9b12bbb None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:34.711493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece01850-0276-4633-8ccd-7004b9b12bbb None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:34.711493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece01850-0276-4633-8ccd-7004b9b12bbb None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:34.712941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 839/1700] 10.222.0.65 () {54 vars in 912 bytes} [Wed Aug 24 03:48:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:48:34.714141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4a5b34-3ae8-4178-b995-beffcc5ff7fb None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:34.718785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4a5b34-3ae8-4178-b995-beffcc5ff7fb None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:34.718785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4a5b34-3ae8-4178-b995-beffcc5ff7fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:34.787368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-fa4a5b34-3ae8-4178-b995-beffcc5ff7fb None None] MFA Rules not processed for user `0772f8b4306f4e239f075f7b1c13790f`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:34.789590 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-fa4a5b34-3ae8-4178-b995-beffcc5ff7fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:34.822655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 862/1701] 10.222.0.65 () {58 vars in 1058 bytes} [Wed Aug 24 03:48:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:48:34.824163 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32bbbe0-b569-4b70-8a1a-95c9eba578f8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:34.826828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32bbbe0-b569-4b70-8a1a-95c9eba578f8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:34.827103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32bbbe0-b569-4b70-8a1a-95c9eba578f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:34.894437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c32bbbe0-b569-4b70-8a1a-95c9eba578f8 None None] MFA Rules not processed for user `64f23f0c8abc406b9700361723623387`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:34.895977 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c32bbbe0-b569-4b70-8a1a-95c9eba578f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:34.918788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 863/1702] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:48:34 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:34.920287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32bbbe0-b569-4b70-8a1a-95c9eba578f8 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:34.951771 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32bbbe0-b569-4b70-8a1a-95c9eba578f8 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:34.961451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d353f021-fa1e-43eb-9960-b11b159f3dd0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:34.964358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d353f021-fa1e-43eb-9960-b11b159f3dd0 service placement] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:34.965023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d353f021-fa1e-43eb-9960-b11b159f3dd0 service placement] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:34.966381 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d353f021-fa1e-43eb-9960-b11b159f3dd0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:34.969949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d353f021-fa1e-43eb-9960-b11b159f3dd0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:34.991115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d353f021-fa1e-43eb-9960-b11b159f3dd0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:34.996864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 840/1703] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:34 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:34.997073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d353f021-fa1e-43eb-9960-b11b159f3dd0 service placement] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:35.001894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4a5b34-3ae8-4178-b995-beffcc5ff7fb None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:35.022889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4a5b34-3ae8-4178-b995-beffcc5ff7fb None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:35.033143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c351000f-0098-4f80-a76b-ec1a03e16a62 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:35.033777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c351000f-0098-4f80-a76b-ec1a03e16a62 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:35.034063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c351000f-0098-4f80-a76b-ec1a03e16a62 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:35.034372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c351000f-0098-4f80-a76b-ec1a03e16a62 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:35.036315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 864/1704] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:35.037885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d353f021-fa1e-43eb-9960-b11b159f3dd0 service placement] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:35.043485 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d353f021-fa1e-43eb-9960-b11b159f3dd0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:35.066246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c351000f-0098-4f80-a76b-ec1a03e16a62 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:35.067744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c351000f-0098-4f80-a76b-ec1a03e16a62 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:35.068694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d353f021-fa1e-43eb-9960-b11b159f3dd0 service placement] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:35.105700 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:35.105993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 841/1705] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:35 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:48:35.107539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:35.107835 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:35.108109 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] PATH_INFO: `/v3/projects/47ed67b528624b85896220c806722c6a` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:35.110211 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=47ed67b528624b85896220c806722c6a)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:35.114225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:35.190077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:INSERT=1,keystone:DELETE=1,keystone:SELECT=98 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:35.213795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c351000f-0098-4f80-a76b-ec1a03e16a62 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:35.216258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c351000f-0098-4f80-a76b-ec1a03e16a62 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:35.256923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c351000f-0098-4f80-a76b-ec1a03e16a62 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:35.284730 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817d1c67-a52d-44dd-8213-629dbf171b75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:35.285504 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817d1c67-a52d-44dd-8213-629dbf171b75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:35.285930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817d1c67-a52d-44dd-8213-629dbf171b75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:35.287489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817d1c67-a52d-44dd-8213-629dbf171b75 admin admin] PATH_INFO: `/v3/users/d5c12c36656a405aa9f1bad5781ecf88` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:35.293248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817d1c67-a52d-44dd-8213-629dbf171b75 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d5c12c36656a405aa9f1bad5781ecf88)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:35.296241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817d1c67-a52d-44dd-8213-629dbf171b75 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:35.391304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47ed67b528624b85896220c806722c6a', 'request_id': 'req-817f5167-d5ed-4004-9841-69c42a957ab3'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:35.401992 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47ed67b528624b85896220c806722c6a', 'request_id': 'req-817f5167-d5ed-4004-9841-69c42a957ab3'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:35.550665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47ed67b528624b85896220c806722c6a', 'request_id': 'req-817f5167-d5ed-4004-9841-69c42a957ab3'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:35.579328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] The token cache is being invalidate because project 47ed67b528624b85896220c806722c6a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:48:35.579877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:35.585749 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:35.590111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 865/1706] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:48:35 2022] DELETE /identity/v3/projects/47ed67b528624b85896220c806722c6a => generated 0 bytes in 553 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:35.597384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:35.604810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:35.662696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-817d1c67-a52d-44dd-8213-629dbf171b75 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd5c12c36656a405aa9f1bad5781ecf88'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:35.665002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-817d1c67-a52d-44dd-8213-629dbf171b75 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd5c12c36656a405aa9f1bad5781ecf88'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:35.687612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-817d1c67-a52d-44dd-8213-629dbf171b75 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd5c12c36656a405aa9f1bad5781ecf88'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:35.734701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-817d1c67-a52d-44dd-8213-629dbf171b75 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd5c12c36656a405aa9f1bad5781ecf88'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:35.787508 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817f5167-d5ed-4004-9841-69c42a957ab3 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:35.800791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:35.801866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:35.802242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:35.802501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] PATH_INFO: `/v3/users/36ac3171e69e4b16946fb1a3d25b1a2b` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:35.804523 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] RBAC: Authorizing `identity:delete_user(user_id=36ac3171e69e4b16946fb1a3d25b1a2b)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:35.807051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:35.817077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-817d1c67-a52d-44dd-8213-629dbf171b75 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd5c12c36656a405aa9f1bad5781ecf88'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:35.826127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 842/1707] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:48:35 2022] DELETE /identity/v3/users/d5c12c36656a405aa9f1bad5781ecf88 => generated 0 bytes in 613 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:48:35.831086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845c21cd-8427-4cb8-9816-d3327441601a None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:35.831621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845c21cd-8427-4cb8-9816-d3327441601a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:35.832721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845c21cd-8427-4cb8-9816-d3327441601a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:35.885790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-845c21cd-8427-4cb8-9816-d3327441601a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:35.887478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-845c21cd-8427-4cb8-9816-d3327441601a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:36.079395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 843/1708] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:36.088337 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '36ac3171e69e4b16946fb1a3d25b1a2b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:36.089465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '36ac3171e69e4b16946fb1a3d25b1a2b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:36.100694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845c21cd-8427-4cb8-9816-d3327441601a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:36.103447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-845c21cd-8427-4cb8-9816-d3327441601a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:36.137610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '36ac3171e69e4b16946fb1a3d25b1a2b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:36.142285 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '36ac3171e69e4b16946fb1a3d25b1a2b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:36.242289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '36ac3171e69e4b16946fb1a3d25b1a2b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:36.264095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845c21cd-8427-4cb8-9816-d3327441601a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:36.276729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588a9ee3-9988-417c-9eab-458322141305 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:36.278107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588a9ee3-9988-417c-9eab-458322141305 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:36.279816 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588a9ee3-9988-417c-9eab-458322141305 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:36.280285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588a9ee3-9988-417c-9eab-458322141305 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:36.284662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 866/1709] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:48:35 2022] DELETE /identity/v3/users/36ac3171e69e4b16946fb1a3d25b1a2b => generated 0 bytes in 688 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:36.285751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-588a9ee3-9988-417c-9eab-458322141305 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:36.307823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588a9ee3-9988-417c-9eab-458322141305 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:36.309886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588a9ee3-9988-417c-9eab-458322141305 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:36.355751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 844/1710] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:36.897886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:37.070445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:37.073560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:37.301062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10526c63-0903-402f-9ea4-d5555ad32b52 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:37.317013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:37.317974 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:37.318569 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:37.319097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] PATH_INFO: `/v3/projects/6252f05139f14716a83b4ea1cd7048ad` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:37.331934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6252f05139f14716a83b4ea1cd7048ad)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:37.333968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:37.543171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6252f05139f14716a83b4ea1cd7048ad', 'request_id': 'req-f76bf6a3-155e-40f9-844d-d16453c11aa6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:37.560459 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6252f05139f14716a83b4ea1cd7048ad', 'request_id': 'req-f76bf6a3-155e-40f9-844d-d16453c11aa6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:37.726604 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6252f05139f14716a83b4ea1cd7048ad', 'request_id': 'req-f76bf6a3-155e-40f9-844d-d16453c11aa6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:37.747345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] The token cache is being invalidate because project 6252f05139f14716a83b4ea1cd7048ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:48:37.748589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:37.749928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:37.752746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 867/1711] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:48:37 2022] DELETE /identity/v3/projects/6252f05139f14716a83b4ea1cd7048ad => generated 0 bytes in 683 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:37.758591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b2fee7-2e71-4e06-9fe8-61dee89d9029 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:37.759565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b2fee7-2e71-4e06-9fe8-61dee89d9029 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:37.760106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b2fee7-2e71-4e06-9fe8-61dee89d9029 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:37.820294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-01b2fee7-2e71-4e06-9fe8-61dee89d9029 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:37.822327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-01b2fee7-2e71-4e06-9fe8-61dee89d9029 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:37.971538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:37.972815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:38.114697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:38.129723 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 845/1712] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:38.137816 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd44c65-6e5e-41cc-8afd-f11cb1b87320 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:38.138296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd44c65-6e5e-41cc-8afd-f11cb1b87320 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:38.138785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd44c65-6e5e-41cc-8afd-f11cb1b87320 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:38.189661 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1fd44c65-6e5e-41cc-8afd-f11cb1b87320 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:38.192100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1fd44c65-6e5e-41cc-8afd-f11cb1b87320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:38.197083 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76bf6a3-155e-40f9-844d-d16453c11aa6 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:38.207968 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:38.208627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:38.213895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 846/1713] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:38.214194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:38.214687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:38.215128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] PATH_INFO: `/v3/projects/43930501ae7e4d2ea740c83c17880de9` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:38.218431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd44c65-6e5e-41cc-8afd-f11cb1b87320 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:38.222819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=43930501ae7e4d2ea740c83c17880de9)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:38.228791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:38.247469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd44c65-6e5e-41cc-8afd-f11cb1b87320 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:38.258932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb45481-a002-4e40-b548-43f886dbbdc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:38.262019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb45481-a002-4e40-b548-43f886dbbdc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:38.262019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb45481-a002-4e40-b548-43f886dbbdc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:38.262019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb45481-a002-4e40-b548-43f886dbbdc8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:38.262019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb45481-a002-4e40-b548-43f886dbbdc8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:38.262019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb45481-a002-4e40-b548-43f886dbbdc8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:38.268892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb45481-a002-4e40-b548-43f886dbbdc8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:38.302954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 847/1714] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:48:38.305403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb45481-a002-4e40-b548-43f886dbbdc8 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:38.330842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb45481-a002-4e40-b548-43f886dbbdc8 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:38.330842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac3a7c0-77a6-489f-a010-278a622aecc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:38.331355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac3a7c0-77a6-489f-a010-278a622aecc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:38.331790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac3a7c0-77a6-489f-a010-278a622aecc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:38.332222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac3a7c0-77a6-489f-a010-278a622aecc5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:38.333008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac3a7c0-77a6-489f-a010-278a622aecc5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:38.336187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac3a7c0-77a6-489f-a010-278a622aecc5 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:38.443825 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43930501ae7e4d2ea740c83c17880de9', 'request_id': 'req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:38.457437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43930501ae7e4d2ea740c83c17880de9', 'request_id': 'req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:38.484792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 848/1715] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:38 2022] POST /identity/v3/projects => generated 371 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:38.486241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655a1844-d611-4595-8887-b9467be36110 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:38.486552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655a1844-d611-4595-8887-b9467be36110 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:38.486908 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655a1844-d611-4595-8887-b9467be36110 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:38.522028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-655a1844-d611-4595-8887-b9467be36110 None None] MFA Rules not processed for user `77b4b56f008c44c081130377cf8ce630`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:38.523905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-655a1844-d611-4595-8887-b9467be36110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:38.618665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43930501ae7e4d2ea740c83c17880de9', 'request_id': 'req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:38.635168 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] The token cache is being invalidate because project 43930501ae7e4d2ea740c83c17880de9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:48:38.635822 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:38.637316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:38.637799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 849/1716] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:38 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:38.643318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655a1844-d611-4595-8887-b9467be36110 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:38.651707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 868/1717] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:48:37 2022] DELETE /identity/v3/projects/43930501ae7e4d2ea740c83c17880de9 => generated 0 bytes in 678 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:38.652813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:38.656840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:38.746032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655a1844-d611-4595-8887-b9467be36110 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:38.752891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168a4426-4570-4030-b520-fc4bccf7e0af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:38.769818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168a4426-4570-4030-b520-fc4bccf7e0af admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:38.772846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168a4426-4570-4030-b520-fc4bccf7e0af admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:38.773466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168a4426-4570-4030-b520-fc4bccf7e0af admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:38.778912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168a4426-4570-4030-b520-fc4bccf7e0af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:38.778912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168a4426-4570-4030-b520-fc4bccf7e0af admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:38.789772 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be69f3a9-46aa-4e81-8ce9-3a4bc5f52ac6 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:38.795682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:38.798771 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:38.798771 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:38.799246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1 admin admin] PATH_INFO: `/v3/projects/a8fe00bed07140dc926803397fbae1bc` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:38.800694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8fe00bed07140dc926803397fbae1bc)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:38.803325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:38.926994 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 850/1718] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:38 2022] POST /identity/v3/users => generated 286 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:38.930464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a2a360-3b05-49d8-8af1-8a2610e1324e None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:38.930464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a2a360-3b05-49d8-8af1-8a2610e1324e None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:38.930464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a2a360-3b05-49d8-8af1-8a2610e1324e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:38.999668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d0a2a360-3b05-49d8-8af1-8a2610e1324e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:38.999668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d0a2a360-3b05-49d8-8af1-8a2610e1324e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:39.011933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 851/1719] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:39.013030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a2a360-3b05-49d8-8af1-8a2610e1324e None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:39.014780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d0a2a360-3b05-49d8-8af1-8a2610e1324e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:39.044130 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a2a360-3b05-49d8-8af1-8a2610e1324e None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:39.060881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8fe00bed07140dc926803397fbae1bc', 'request_id': 'req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:39.083619 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8fe00bed07140dc926803397fbae1bc', 'request_id': 'req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:39.223069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8fe00bed07140dc926803397fbae1bc', 'request_id': 'req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:39.248056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1 admin admin] The token cache is being invalidate because project a8fe00bed07140dc926803397fbae1bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:48:39.253241 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:39.260647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-bfd9c40f-7cea-45af-8a06-8f755ff6e9e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:39.273442 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 869/1720] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:48:38 2022] DELETE /identity/v3/projects/a8fe00bed07140dc926803397fbae1bc => generated 0 bytes in 621 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:39.278963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c097e7-fd47-4386-a78b-8d0730f14122 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:39.288591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c097e7-fd47-4386-a78b-8d0730f14122 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:39.296911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c097e7-fd47-4386-a78b-8d0730f14122 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:39.308574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99475cb-7455-4648-893e-47eef47881cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:39.309213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99475cb-7455-4648-893e-47eef47881cb admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:39.309489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99475cb-7455-4648-893e-47eef47881cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:39.314945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99475cb-7455-4648-893e-47eef47881cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:39.315688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99475cb-7455-4648-893e-47eef47881cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:39.322058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99475cb-7455-4648-893e-47eef47881cb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:39.370688 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e6c097e7-fd47-4386-a78b-8d0730f14122 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:39.370777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 852/1721] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:39 2022] GET /identity/v3/roles => generated 1420 bytes in 355 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:39.370777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99475cb-7455-4648-893e-47eef47881cb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:39.372148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e6c097e7-fd47-4386-a78b-8d0730f14122 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:39.378747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-e99475cb-7455-4648-893e-47eef47881cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:39.417001 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 870/1722] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:39.418082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c097e7-fd47-4386-a78b-8d0730f14122 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:39.426139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e6c097e7-fd47-4386-a78b-8d0730f14122 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:39.433717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99475cb-7455-4648-893e-47eef47881cb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:39.451810 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e87f707-12cf-4ca9-9fdb-e170fcd8bb4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:39.465275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e87f707-12cf-4ca9-9fdb-e170fcd8bb4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:39.466755 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e87f707-12cf-4ca9-9fdb-e170fcd8bb4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:39.467228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e87f707-12cf-4ca9-9fdb-e170fcd8bb4d admin admin] PATH_INFO: `/v3/users/651b943399c34c0f933ea32b7e1ed065` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:39.473353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e87f707-12cf-4ca9-9fdb-e170fcd8bb4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=651b943399c34c0f933ea32b7e1ed065)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:39.475720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c097e7-fd47-4386-a78b-8d0730f14122 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:39.485834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e87f707-12cf-4ca9-9fdb-e170fcd8bb4d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:39.496163 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e16776-581d-4728-a4d1-bb41b6d3c671 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:39.497767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e16776-581d-4728-a4d1-bb41b6d3c671 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:39.499099 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e16776-581d-4728-a4d1-bb41b6d3c671 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:39.499546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e16776-581d-4728-a4d1-bb41b6d3c671 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:39.500399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e16776-581d-4728-a4d1-bb41b6d3c671 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:39.505597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e16776-581d-4728-a4d1-bb41b6d3c671 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:39.560314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 871/1723] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:39 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:39.562757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e16776-581d-4728-a4d1-bb41b6d3c671 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:39.588427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e16776-581d-4728-a4d1-bb41b6d3c671 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:39.629872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59beb36f-29d5-4879-a5e5-d6e90a092ff5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:39.631844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59beb36f-29d5-4879-a5e5-d6e90a092ff5 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:39.631844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59beb36f-29d5-4879-a5e5-d6e90a092ff5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:39.632070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59beb36f-29d5-4879-a5e5-d6e90a092ff5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:39.633094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59beb36f-29d5-4879-a5e5-d6e90a092ff5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:39.633410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59beb36f-29d5-4879-a5e5-d6e90a092ff5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:39.635221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59beb36f-29d5-4879-a5e5-d6e90a092ff5 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:39.655910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 872/1724] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:48:39.656797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59beb36f-29d5-4879-a5e5-d6e90a092ff5 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:39.674317 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59beb36f-29d5-4879-a5e5-d6e90a092ff5 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:39.682332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f7434e-86bf-4281-abd6-a448b7a1ad97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:39.683873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f7434e-86bf-4281-abd6-a448b7a1ad97 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:39.684214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f7434e-86bf-4281-abd6-a448b7a1ad97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:39.684528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f7434e-86bf-4281-abd6-a448b7a1ad97 admin admin] PATH_INFO: `/v3/projects/5d3925a8b5fe44868a8db62237ea42ef/users/1d05c440cf484828b3589a671f69b503/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:39.735132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f7434e-86bf-4281-abd6-a448b7a1ad97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d3925a8b5fe44868a8db62237ea42ef, user_id=1d05c440cf484828b3589a671f69b503, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:39.741351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f7434e-86bf-4281-abd6-a448b7a1ad97 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:39.866801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0e87f707-12cf-4ca9-9fdb-e170fcd8bb4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '651b943399c34c0f933ea32b7e1ed065'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:39.869342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0e87f707-12cf-4ca9-9fdb-e170fcd8bb4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '651b943399c34c0f933ea32b7e1ed065'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:39.888548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 873/1725] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:39 2022] PUT /identity/v3/projects/5d3925a8b5fe44868a8db62237ea42ef/users/1d05c440cf484828b3589a671f69b503/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:39.889612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f7434e-86bf-4281-abd6-a448b7a1ad97 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:39.919001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0e87f707-12cf-4ca9-9fdb-e170fcd8bb4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '651b943399c34c0f933ea32b7e1ed065'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:39.948008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0e87f707-12cf-4ca9-9fdb-e170fcd8bb4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '651b943399c34c0f933ea32b7e1ed065'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:40.083510 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0e87f707-12cf-4ca9-9fdb-e170fcd8bb4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '651b943399c34c0f933ea32b7e1ed065'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:40.092479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f7434e-86bf-4281-abd6-a448b7a1ad97 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:40.099951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cb297c-2888-4ae1-8540-f0e34272d66f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:40.101658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cb297c-2888-4ae1-8540-f0e34272d66f admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:40.102111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cb297c-2888-4ae1-8540-f0e34272d66f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:40.102560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cb297c-2888-4ae1-8540-f0e34272d66f admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:40.103380 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cb297c-2888-4ae1-8540-f0e34272d66f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:40.105580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cb297c-2888-4ae1-8540-f0e34272d66f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:40.110507 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 853/1726] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:48:39 2022] DELETE /identity/v3/users/651b943399c34c0f933ea32b7e1ed065 => generated 0 bytes in 742 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:40.125541 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0a31ba-a94d-4c51-82c9-2c29d1bb09e7 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:40.128935 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0a31ba-a94d-4c51-82c9-2c29d1bb09e7 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:40.129442 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0a31ba-a94d-4c51-82c9-2c29d1bb09e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:40.188485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-fb0a31ba-a94d-4c51-82c9-2c29d1bb09e7 None None] MFA Rules not processed for user `1d05c440cf484828b3589a671f69b503`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:40.191474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-fb0a31ba-a94d-4c51-82c9-2c29d1bb09e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:40.290581 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 874/1727] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:39 2022] POST /identity/v3/projects => generated 393 bytes in 396 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:40.293333 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cb297c-2888-4ae1-8540-f0e34272d66f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:40.297269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-30cb297c-2888-4ae1-8540-f0e34272d66f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:40.363590 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 854/1728] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:40 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:40.364991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0a31ba-a94d-4c51-82c9-2c29d1bb09e7 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:40.373223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-fb0a31ba-a94d-4c51-82c9-2c29d1bb09e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:40.526794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cb297c-2888-4ae1-8540-f0e34272d66f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:40.541785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027026b5-670a-4620-9689-a05e3aaea25e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:40.542834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027026b5-670a-4620-9689-a05e3aaea25e admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:40.543298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027026b5-670a-4620-9689-a05e3aaea25e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:40.543766 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027026b5-670a-4620-9689-a05e3aaea25e admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:40.544585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027026b5-670a-4620-9689-a05e3aaea25e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:40.550692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027026b5-670a-4620-9689-a05e3aaea25e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:40.596576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0a31ba-a94d-4c51-82c9-2c29d1bb09e7 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:40.618453 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee4172b-8ae2-4850-8295-cad6af4f9e20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:40.619054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee4172b-8ae2-4850-8295-cad6af4f9e20 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:40.619187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee4172b-8ae2-4850-8295-cad6af4f9e20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:40.619452 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee4172b-8ae2-4850-8295-cad6af4f9e20 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:40.620152 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee4172b-8ae2-4850-8295-cad6af4f9e20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:40.635081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee4172b-8ae2-4850-8295-cad6af4f9e20 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:40.671814 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 875/1729] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:40 2022] POST /identity/v3/users => generated 297 bytes in 380 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:40.673418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4139f38f-2e2f-4032-8058-131448dec883 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:40.673709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4139f38f-2e2f-4032-8058-131448dec883 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:40.674001 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4139f38f-2e2f-4032-8058-131448dec883 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:40.700221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-4139f38f-2e2f-4032-8058-131448dec883 None None] MFA Rules not processed for user `bd8d967d9ecb410ea47e130ac3c8b4a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:40.703519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4139f38f-2e2f-4032-8058-131448dec883 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:40.832763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 876/1730] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:40 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:40.834514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4139f38f-2e2f-4032-8058-131448dec883 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:40.856662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4139f38f-2e2f-4032-8058-131448dec883 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:40.863009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79660d1f-853d-4aa5-9b79-e45519a9b9fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:40.863996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79660d1f-853d-4aa5-9b79-e45519a9b9fa admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:40.864134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79660d1f-853d-4aa5-9b79-e45519a9b9fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:40.864316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79660d1f-853d-4aa5-9b79-e45519a9b9fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:40.869670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79660d1f-853d-4aa5-9b79-e45519a9b9fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:40.870119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:40.883502 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79660d1f-853d-4aa5-9b79-e45519a9b9fa admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:40.903499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 877/1731] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:40 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:40.908275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79660d1f-853d-4aa5-9b79-e45519a9b9fa admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:40.950871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79660d1f-853d-4aa5-9b79-e45519a9b9fa admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:40.959422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 855/1732] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:40 2022] POST /identity/v3/projects => generated 371 bytes in 598 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:40.964076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee4172b-8ae2-4850-8295-cad6af4f9e20 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:40.975722 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5964fe07-0419-4730-97b4-a07a68c3e1b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:40.976671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5964fe07-0419-4730-97b4-a07a68c3e1b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:40.977095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5964fe07-0419-4730-97b4-a07a68c3e1b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:40.977533 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5964fe07-0419-4730-97b4-a07a68c3e1b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:40.978397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5964fe07-0419-4730-97b4-a07a68c3e1b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:40.992127 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5964fe07-0419-4730-97b4-a07a68c3e1b7 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:41.022899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 878/1733] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:40 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:48:41.027226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5964fe07-0419-4730-97b4-a07a68c3e1b7 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:41.046596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5964fe07-0419-4730-97b4-a07a68c3e1b7 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:41.097684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee4172b-8ae2-4850-8295-cad6af4f9e20 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:41.103964 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0637040f-8abe-4a2e-a877-713884843ba2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:41.104589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0637040f-8abe-4a2e-a877-713884843ba2 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:41.104910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0637040f-8abe-4a2e-a877-713884843ba2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:41.105182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0637040f-8abe-4a2e-a877-713884843ba2 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:41.105874 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0637040f-8abe-4a2e-a877-713884843ba2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:41.109785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0637040f-8abe-4a2e-a877-713884843ba2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:41.160727 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:41.184070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc22dac-749a-453f-87ef-f4b3c34092ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:41.184966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc22dac-749a-453f-87ef-f4b3c34092ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:41.185246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc22dac-749a-453f-87ef-f4b3c34092ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:41.185691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc22dac-749a-453f-87ef-f4b3c34092ba admin admin] PATH_INFO: `/v3/projects/c1e6159c2bba4176913d8775ae1f227e/users/a7e2e869af8143339399b2e5e9d29548/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:41.209982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc22dac-749a-453f-87ef-f4b3c34092ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1e6159c2bba4176913d8775ae1f227e, user_id=a7e2e869af8143339399b2e5e9d29548, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:41.222646 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc22dac-749a-453f-87ef-f4b3c34092ba admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:41.244171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 856/1734] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:40 2022] POST /identity/v3/users => generated 292 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:41.249373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0637040f-8abe-4a2e-a877-713884843ba2 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:41.280202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0637040f-8abe-4a2e-a877-713884843ba2 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:41.293676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f403f119-19fa-4fa4-9dd6-7f0dab94de3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:41.296251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f403f119-19fa-4fa4-9dd6-7f0dab94de3a admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:41.297058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f403f119-19fa-4fa4-9dd6-7f0dab94de3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:41.306843 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f403f119-19fa-4fa4-9dd6-7f0dab94de3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:41.326776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f403f119-19fa-4fa4-9dd6-7f0dab94de3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:41.333043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f403f119-19fa-4fa4-9dd6-7f0dab94de3a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:41.363464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 857/1735] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:41 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:48:41.376531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 879/1736] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:41 2022] PUT /identity/v3/projects/c1e6159c2bba4176913d8775ae1f227e/users/a7e2e869af8143339399b2e5e9d29548/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:48:41.377917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc22dac-749a-453f-87ef-f4b3c34092ba admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:41.379175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f403f119-19fa-4fa4-9dd6-7f0dab94de3a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:41.384296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f403f119-19fa-4fa4-9dd6-7f0dab94de3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:41.467107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f403f119-19fa-4fa4-9dd6-7f0dab94de3a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:41.608845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc22dac-749a-453f-87ef-f4b3c34092ba admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:41.626471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cd56cd-5072-4e6b-b2ea-c9b8de3f086a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:41.627551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cd56cd-5072-4e6b-b2ea-c9b8de3f086a admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:41.627838 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cd56cd-5072-4e6b-b2ea-c9b8de3f086a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:41.628256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cd56cd-5072-4e6b-b2ea-c9b8de3f086a admin admin] PATH_INFO: `/v3/projects/d8ee3b88c6564ba6901d88e720d32203/users/de2ae72bc3b44c489efd46390f796bce/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:41.707743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:41.712992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:41.713675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:41.718030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] PATH_INFO: `/v3/projects/eb3f65c2c4b2464893d90fbab9dd7968` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:41.719607 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cd56cd-5072-4e6b-b2ea-c9b8de3f086a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d8ee3b88c6564ba6901d88e720d32203, user_id=de2ae72bc3b44c489efd46390f796bce, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:41.721212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb3f65c2c4b2464893d90fbab9dd7968)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:41.722404 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cd56cd-5072-4e6b-b2ea-c9b8de3f086a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:41.724557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:41.907062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 880/1737] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:41 2022] PUT /identity/v3/projects/d8ee3b88c6564ba6901d88e720d32203/users/de2ae72bc3b44c489efd46390f796bce/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 529 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:41.908627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9ca726-60a0-41a8-86bc-87c79d3d78ec None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:41.913196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9ca726-60a0-41a8-86bc-87c79d3d78ec None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:41.913856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9ca726-60a0-41a8-86bc-87c79d3d78ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:41.971340 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb3f65c2c4b2464893d90fbab9dd7968', 'request_id': 'req-e32bd169-d216-4fe9-9815-cdf5862fe97e'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:41.992420 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-0e9ca726-60a0-41a8-86bc-87c79d3d78ec None None] MFA Rules not processed for user `a7e2e869af8143339399b2e5e9d29548`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:41.994585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0e9ca726-60a0-41a8-86bc-87c79d3d78ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:42.005866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb3f65c2c4b2464893d90fbab9dd7968', 'request_id': 'req-e32bd169-d216-4fe9-9815-cdf5862fe97e'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:42.036415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb3f65c2c4b2464893d90fbab9dd7968', 'request_id': 'req-e32bd169-d216-4fe9-9815-cdf5862fe97e'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:42.183427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 881/1738] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:41 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:42.184426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9ca726-60a0-41a8-86bc-87c79d3d78ec None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:42.263922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] The token cache is being invalidate because project eb3f65c2c4b2464893d90fbab9dd7968 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:48:42.263922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:42.263922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:42.267448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 858/1739] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:48:41 2022] DELETE /identity/v3/projects/eb3f65c2c4b2464893d90fbab9dd7968 => generated 0 bytes in 902 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:42.268328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:42.284711 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:42.319645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9ca726-60a0-41a8-86bc-87c79d3d78ec None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:42.337691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5381350-39cb-44f8-b773-4089491c5619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:42.338718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5381350-39cb-44f8-b773-4089491c5619 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:42.339219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5381350-39cb-44f8-b773-4089491c5619 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:42.339711 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5381350-39cb-44f8-b773-4089491c5619 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:42.340573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5381350-39cb-44f8-b773-4089491c5619 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:42.342463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5381350-39cb-44f8-b773-4089491c5619 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:42.365181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 882/1740] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:42 2022] GET /identity/v3/roles => generated 1420 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:42.372831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b5a3a6-fe49-4929-aa74-c625f1c90bdc None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:42.373416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b5a3a6-fe49-4929-aa74-c625f1c90bdc None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:42.375355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b5a3a6-fe49-4929-aa74-c625f1c90bdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:42.499459 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e8b5a3a6-fe49-4929-aa74-c625f1c90bdc None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:42.501241 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e8b5a3a6-fe49-4929-aa74-c625f1c90bdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:42.583189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32bd169-d216-4fe9-9815-cdf5862fe97e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:42.594852 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be67ce6-dc96-4186-aee6-aa789c47db03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:42.595501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be67ce6-dc96-4186-aee6-aa789c47db03 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:42.595782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be67ce6-dc96-4186-aee6-aa789c47db03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:42.596080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be67ce6-dc96-4186-aee6-aa789c47db03 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:42.596991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be67ce6-dc96-4186-aee6-aa789c47db03 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:42.603245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be67ce6-dc96-4186-aee6-aa789c47db03 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:42.682866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 883/1741] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:48:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 317 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:42.685712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1682ac4a-15af-4d29-b1e5-94ec83e97b94 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:42.689793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1682ac4a-15af-4d29-b1e5-94ec83e97b94 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:42.691002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1682ac4a-15af-4d29-b1e5-94ec83e97b94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:42.740074 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1682ac4a-15af-4d29-b1e5-94ec83e97b94 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:42.748677 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1682ac4a-15af-4d29-b1e5-94ec83e97b94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:42.774565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 884/1742] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:42.776205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1682ac4a-15af-4d29-b1e5-94ec83e97b94 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:42.779776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1682ac4a-15af-4d29-b1e5-94ec83e97b94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:42.797779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1682ac4a-15af-4d29-b1e5-94ec83e97b94 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:42.805724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2465cba8-2cc3-4302-858b-88a4999ad531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:42.806801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2465cba8-2cc3-4302-858b-88a4999ad531 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:42.807423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2465cba8-2cc3-4302-858b-88a4999ad531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:42.807885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2465cba8-2cc3-4302-858b-88a4999ad531 admin admin] PATH_INFO: `/v3/projects/d8ee3b88c6564ba6901d88e720d32203/users/de2ae72bc3b44c489efd46390f796bce/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:42.811529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2465cba8-2cc3-4302-858b-88a4999ad531 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d8ee3b88c6564ba6901d88e720d32203, user_id=de2ae72bc3b44c489efd46390f796bce, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:42.835138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2465cba8-2cc3-4302-858b-88a4999ad531 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:42.852968 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 859/1743] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:42 2022] POST /identity/v3/projects => generated 391 bytes in 585 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:42.860022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2243c9-c3f3-4467-aee1-273f69ab00e4 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:42.860400 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2243c9-c3f3-4467-aee1-273f69ab00e4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:42.860707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2243c9-c3f3-4467-aee1-273f69ab00e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:42.863630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 885/1744] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:42 2022] PUT /identity/v3/projects/d8ee3b88c6564ba6901d88e720d32203/users/de2ae72bc3b44c489efd46390f796bce/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:42.864757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2465cba8-2cc3-4302-858b-88a4999ad531 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:42.933066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-3d2243c9-c3f3-4467-aee1-273f69ab00e4 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:42.937138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3d2243c9-c3f3-4467-aee1-273f69ab00e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:43.138843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2465cba8-2cc3-4302-858b-88a4999ad531 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:43.151566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78db73c7-1639-45fc-a3cd-94083f60a841 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:43.168435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78db73c7-1639-45fc-a3cd-94083f60a841 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:43.168435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78db73c7-1639-45fc-a3cd-94083f60a841 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:43.168435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78db73c7-1639-45fc-a3cd-94083f60a841 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:43.168435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78db73c7-1639-45fc-a3cd-94083f60a841 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:43.168435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78db73c7-1639-45fc-a3cd-94083f60a841 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:43.200770 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 860/1745] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 343 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:43.204547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a846b3-d911-41d8-98f6-b1f2c8f582da None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:43.205290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a846b3-d911-41d8-98f6-b1f2c8f582da None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:43.205729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a846b3-d911-41d8-98f6-b1f2c8f582da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:43.221065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:43.265934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-16a846b3-d911-41d8-98f6-b1f2c8f582da None None] MFA Rules not processed for user `de2ae72bc3b44c489efd46390f796bce`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:43.267916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-16a846b3-d911-41d8-98f6-b1f2c8f582da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:43.391819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 886/1746] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:42 2022] POST /identity/v3/users => generated 302 bytes in 527 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:43.393916 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6991d70-5227-4250-ae90-42de53283bf3 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:43.395811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6991d70-5227-4250-ae90-42de53283bf3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:43.396430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6991d70-5227-4250-ae90-42de53283bf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:43.448742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 861/1747] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:43 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:43.451437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a846b3-d911-41d8-98f6-b1f2c8f582da None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:43.484850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e6991d70-5227-4250-ae90-42de53283bf3 None None] MFA Rules not processed for user `7f4991f9c6ca4fc99f756f75f9460808`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:43.486709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e6991d70-5227-4250-ae90-42de53283bf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:43.509482 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a846b3-d911-41d8-98f6-b1f2c8f582da None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:43.515188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224c2186-2c78-49b9-8ee7-1490d7133224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:43.516146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224c2186-2c78-49b9-8ee7-1490d7133224 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:43.516910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224c2186-2c78-49b9-8ee7-1490d7133224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:43.517378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224c2186-2c78-49b9-8ee7-1490d7133224 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:43.518303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224c2186-2c78-49b9-8ee7-1490d7133224 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:48:43.518799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224c2186-2c78-49b9-8ee7-1490d7133224 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:43.522639 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224c2186-2c78-49b9-8ee7-1490d7133224 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:43.550252 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 862/1748] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:48:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:48:43.552406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224c2186-2c78-49b9-8ee7-1490d7133224 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:43.554995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-224c2186-2c78-49b9-8ee7-1490d7133224 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:43.728078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 887/1749] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 334 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:43.731465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6991d70-5227-4250-ae90-42de53283bf3 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:43.746835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224c2186-2c78-49b9-8ee7-1490d7133224 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:43.757783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6991d70-5227-4250-ae90-42de53283bf3 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:43.759168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8b6333-165a-4e21-8959-0b256ae0f7cc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:43.760098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8b6333-165a-4e21-8959-0b256ae0f7cc service swift] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:43.760496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8b6333-165a-4e21-8959-0b256ae0f7cc service swift] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:43.760777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8b6333-165a-4e21-8959-0b256ae0f7cc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:43.769913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845a7d4c-daf9-465a-84f9-35f8c8b55f16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:43.771077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845a7d4c-daf9-465a-84f9-35f8c8b55f16 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:43.774239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845a7d4c-daf9-465a-84f9-35f8c8b55f16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:43.775132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845a7d4c-daf9-465a-84f9-35f8c8b55f16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:43.775994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845a7d4c-daf9-465a-84f9-35f8c8b55f16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:43.779257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845a7d4c-daf9-465a-84f9-35f8c8b55f16 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:43.820926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 888/1750] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:43 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:43.824571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845a7d4c-daf9-465a-84f9-35f8c8b55f16 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:43.828783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-845a7d4c-daf9-465a-84f9-35f8c8b55f16 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:43.862665 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8b6333-165a-4e21-8959-0b256ae0f7cc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:43.864281 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8b6333-165a-4e21-8959-0b256ae0f7cc service swift] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:43.930726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 863/1751] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:48:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:48:43.931681 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8b6333-165a-4e21-8959-0b256ae0f7cc service swift] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:43.967009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:43.973807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8b6333-165a-4e21-8959-0b256ae0f7cc service swift] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:43.987054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df83428-6e80-4fc1-be57-31c6d9acf6b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:43.987690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df83428-6e80-4fc1-be57-31c6d9acf6b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:43.994432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df83428-6e80-4fc1-be57-31c6d9acf6b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:43.994432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df83428-6e80-4fc1-be57-31c6d9acf6b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:43.994432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df83428-6e80-4fc1-be57-31c6d9acf6b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:43.994432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df83428-6e80-4fc1-be57-31c6d9acf6b5 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:44.052156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845a7d4c-daf9-465a-84f9-35f8c8b55f16 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:44.052156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7217bcc8-21d4-4bec-9104-176879d50e41 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:44.052156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7217bcc8-21d4-4bec-9104-176879d50e41 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:44.052156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7217bcc8-21d4-4bec-9104-176879d50e41 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:44.052463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7217bcc8-21d4-4bec-9104-176879d50e41 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:44.053491 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7217bcc8-21d4-4bec-9104-176879d50e41 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:44.055895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7217bcc8-21d4-4bec-9104-176879d50e41 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:44.073211 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 889/1752] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:43 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:44.075614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ead0b1-7c82-40b5-807c-8e0ebe378a71 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:44.076277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ead0b1-7c82-40b5-807c-8e0ebe378a71 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:44.077122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ead0b1-7c82-40b5-807c-8e0ebe378a71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:44.122752 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-62ead0b1-7c82-40b5-807c-8e0ebe378a71 None None] MFA Rules not processed for user `64f23f0c8abc406b9700361723623387`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:44.125727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-62ead0b1-7c82-40b5-807c-8e0ebe378a71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:44.211206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 864/1753] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:43 2022] POST /identity/v3/projects => generated 365 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:44.213163 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df83428-6e80-4fc1-be57-31c6d9acf6b5 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:44.311459 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 890/1754] 10.222.0.71 () {58 vars in 1015 bytes} [Wed Aug 24 03:48:44 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:44.337812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ead0b1-7c82-40b5-807c-8e0ebe378a71 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:44.368177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ead0b1-7c82-40b5-807c-8e0ebe378a71 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:44.439277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df83428-6e80-4fc1-be57-31c6d9acf6b5 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:44.451190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79a7c6f-549b-4e0a-ac09-35d41b084b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:44.452000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79a7c6f-549b-4e0a-ac09-35d41b084b88 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:44.461776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79a7c6f-549b-4e0a-ac09-35d41b084b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:44.461776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79a7c6f-549b-4e0a-ac09-35d41b084b88 admin admin] PATH_INFO: `/v3/projects/bdf427d8d68f47098907e9b7f92669cc/users/71b666a069b4402e96a6898d76a12f94/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:44.510320 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeaa8a6-fa4f-4dff-a2ce-1f07b28e5d0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:44.511906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeaa8a6-fa4f-4dff-a2ce-1f07b28e5d0a admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:44.512345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeaa8a6-fa4f-4dff-a2ce-1f07b28e5d0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:44.512942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeaa8a6-fa4f-4dff-a2ce-1f07b28e5d0a admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:44.513750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adeaa8a6-fa4f-4dff-a2ce-1f07b28e5d0a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:44.516183 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adeaa8a6-fa4f-4dff-a2ce-1f07b28e5d0a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:44.528223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79a7c6f-549b-4e0a-ac09-35d41b084b88 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bdf427d8d68f47098907e9b7f92669cc, user_id=71b666a069b4402e96a6898d76a12f94, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:44.535321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79a7c6f-549b-4e0a-ac09-35d41b084b88 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:44.738204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 891/1755] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:44 2022] POST /identity/v3/users => generated 283 bytes in 426 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:44.739750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeaa8a6-fa4f-4dff-a2ce-1f07b28e5d0a admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:44.746946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-adeaa8a6-fa4f-4dff-a2ce-1f07b28e5d0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:44.836576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 865/1756] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:44 2022] PUT /identity/v3/projects/bdf427d8d68f47098907e9b7f92669cc/users/71b666a069b4402e96a6898d76a12f94/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 624 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:44.837585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79a7c6f-549b-4e0a-ac09-35d41b084b88 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:44.937323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeaa8a6-fa4f-4dff-a2ce-1f07b28e5d0a admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:45.022525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79a7c6f-549b-4e0a-ac09-35d41b084b88 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:45.028600 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0453bfbf-a7ef-420d-aac9-74869a9e6503 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:45.030375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0453bfbf-a7ef-420d-aac9-74869a9e6503 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:45.030728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0453bfbf-a7ef-420d-aac9-74869a9e6503 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:45.031010 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0453bfbf-a7ef-420d-aac9-74869a9e6503 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:45.039144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0453bfbf-a7ef-420d-aac9-74869a9e6503 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:45.088176 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41ff27a-38f3-4697-8b8d-06d836770b0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:45.089157 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41ff27a-38f3-4697-8b8d-06d836770b0b service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:45.089609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41ff27a-38f3-4697-8b8d-06d836770b0b service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:45.090054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41ff27a-38f3-4697-8b8d-06d836770b0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:45.129940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41ff27a-38f3-4697-8b8d-06d836770b0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:45.135185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41ff27a-38f3-4697-8b8d-06d836770b0b service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:45.174923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 892/1757] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:45.184216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41ff27a-38f3-4697-8b8d-06d836770b0b service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:45.204334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0453bfbf-a7ef-420d-aac9-74869a9e6503 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:45.208687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0453bfbf-a7ef-420d-aac9-74869a9e6503 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:45.258667 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 866/1758] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:44 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:45.262711 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0453bfbf-a7ef-420d-aac9-74869a9e6503 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:45.428870 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41ff27a-38f3-4697-8b8d-06d836770b0b service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:45.469332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d5f9f5-3805-4d0a-bafc-fc1a7505d8e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:45.469988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d5f9f5-3805-4d0a-bafc-fc1a7505d8e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:45.484244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0453bfbf-a7ef-420d-aac9-74869a9e6503 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:45.528877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d5f9f5-3805-4d0a-bafc-fc1a7505d8e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:45.528877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d5f9f5-3805-4d0a-bafc-fc1a7505d8e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:45.528877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d5f9f5-3805-4d0a-bafc-fc1a7505d8e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:45.528877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d5f9f5-3805-4d0a-bafc-fc1a7505d8e1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:45.530139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a42b13a-b93d-46e1-8bc3-c7bcc041701a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:45.530139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a42b13a-b93d-46e1-8bc3-c7bcc041701a admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:45.530139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a42b13a-b93d-46e1-8bc3-c7bcc041701a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:45.530139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a42b13a-b93d-46e1-8bc3-c7bcc041701a admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:45.530917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a42b13a-b93d-46e1-8bc3-c7bcc041701a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:45.530917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a42b13a-b93d-46e1-8bc3-c7bcc041701a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:45.530917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 867/1759] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:45.531705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a42b13a-b93d-46e1-8bc3-c7bcc041701a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:45.535597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 893/1760] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 360 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:45.536573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d5f9f5-3805-4d0a-bafc-fc1a7505d8e1 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:45.564828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a42b13a-b93d-46e1-8bc3-c7bcc041701a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:45.574813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d15dff-4e2a-48fb-81cd-f91f57049ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:45.575079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d5f9f5-3805-4d0a-bafc-fc1a7505d8e1 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:45.575961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d15dff-4e2a-48fb-81cd-f91f57049ee7 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:45.576880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d15dff-4e2a-48fb-81cd-f91f57049ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:45.577527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d15dff-4e2a-48fb-81cd-f91f57049ee7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:45.579321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d15dff-4e2a-48fb-81cd-f91f57049ee7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:45.584001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d15dff-4e2a-48fb-81cd-f91f57049ee7 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:45.601254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab3233a-654d-407c-8756-ca1b20f43d9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:45.604233 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab3233a-654d-407c-8756-ca1b20f43d9e admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:45.604493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab3233a-654d-407c-8756-ca1b20f43d9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:45.604753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab3233a-654d-407c-8756-ca1b20f43d9e admin admin] PATH_INFO: `/v3/projects/bdf427d8d68f47098907e9b7f92669cc/users/71b666a069b4402e96a6898d76a12f94/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:45.614140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab3233a-654d-407c-8756-ca1b20f43d9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=bdf427d8d68f47098907e9b7f92669cc, user_id=71b666a069b4402e96a6898d76a12f94, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:45.623019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab3233a-654d-407c-8756-ca1b20f43d9e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:45.633335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 868/1761] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:45 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:45.644784 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 894/1762] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:45 2022] PUT /identity/v3/projects/bdf427d8d68f47098907e9b7f92669cc/users/71b666a069b4402e96a6898d76a12f94/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:45.656883 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d15dff-4e2a-48fb-81cd-f91f57049ee7 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:45.662396 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b9a57c-94fa-43fc-8c1d-d1eec2f897e8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:45.663378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b9a57c-94fa-43fc-8c1d-d1eec2f897e8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:45.663736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b9a57c-94fa-43fc-8c1d-d1eec2f897e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:45.693932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d15dff-4e2a-48fb-81cd-f91f57049ee7 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:45.818672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e0b9a57c-94fa-43fc-8c1d-d1eec2f897e8 None None] MFA Rules not processed for user `71b666a069b4402e96a6898d76a12f94`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:45.820243 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e0b9a57c-94fa-43fc-8c1d-d1eec2f897e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:45.828394 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf0b58b-fb5f-4fc9-bdd7-1e5ed17a23af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:45.829408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf0b58b-fb5f-4fc9-bdd7-1e5ed17a23af admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:45.830094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf0b58b-fb5f-4fc9-bdd7-1e5ed17a23af admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:45.830585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf0b58b-fb5f-4fc9-bdd7-1e5ed17a23af admin admin] PATH_INFO: `/v3/projects/20293ee4d4fe4015b2c040955e474d1a/users/792280f0e991489d8273fad8d01f1fee/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:45.871016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf0b58b-fb5f-4fc9-bdd7-1e5ed17a23af admin admin] RBAC: Authorizing `identity:create_grant(project_id=20293ee4d4fe4015b2c040955e474d1a, user_id=792280f0e991489d8273fad8d01f1fee, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:45.882886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf0b58b-fb5f-4fc9-bdd7-1e5ed17a23af admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:46.083475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 869/1763] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:45 2022] PUT /identity/v3/projects/20293ee4d4fe4015b2c040955e474d1a/users/792280f0e991489d8273fad8d01f1fee/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:46.084282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 895/1764] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:45 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 423 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:46.091102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb95fb96-ce1a-4a5b-a150-fcd1b50369af None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:46.092529 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb95fb96-ce1a-4a5b-a150-fcd1b50369af None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:46.092817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb95fb96-ce1a-4a5b-a150-fcd1b50369af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:46.097527 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b9a57c-94fa-43fc-8c1d-d1eec2f897e8 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:46.099031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e0b9a57c-94fa-43fc-8c1d-d1eec2f897e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:46.158559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-bb95fb96-ce1a-4a5b-a150-fcd1b50369af None None] MFA Rules not processed for user `792280f0e991489d8273fad8d01f1fee`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:46.159877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-bb95fb96-ce1a-4a5b-a150-fcd1b50369af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:46.302614 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 870/1765] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:46 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:46.304173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b9a57c-94fa-43fc-8c1d-d1eec2f897e8 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:46.308112 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb95fb96-ce1a-4a5b-a150-fcd1b50369af None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:46.407533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:46.466751 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a202633-0538-4b33-a7ed-e30d69d8f18a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:46.468339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a202633-0538-4b33-a7ed-e30d69d8f18a service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:46.468964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a202633-0538-4b33-a7ed-e30d69d8f18a service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:46.469403 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a202633-0538-4b33-a7ed-e30d69d8f18a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:46.500501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb95fb96-ce1a-4a5b-a150-fcd1b50369af None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:46.506252 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1ddddb-b1c7-47d3-b7a6-823726af913c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:46.506912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1ddddb-b1c7-47d3-b7a6-823726af913c admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:46.507191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1ddddb-b1c7-47d3-b7a6-823726af913c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:46.507467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1ddddb-b1c7-47d3-b7a6-823726af913c admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:46.508152 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1ddddb-b1c7-47d3-b7a6-823726af913c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:46.510186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1ddddb-b1c7-47d3-b7a6-823726af913c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:46.634908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a202633-0538-4b33-a7ed-e30d69d8f18a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:46.639608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a202633-0538-4b33-a7ed-e30d69d8f18a service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:46.672085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 896/1766] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:46 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 575 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:48:46.690791 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 871/1767] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:48:46 2022] POST /identity/v3/projects => generated 365 bytes in 385 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:46.699627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a202633-0538-4b33-a7ed-e30d69d8f18a service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:46.732626 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a202633-0538-4b33-a7ed-e30d69d8f18a service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:46.816935 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1ddddb-b1c7-47d3-b7a6-823726af913c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:46.916835 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642a17c7-58e9-4b76-bdf6-70bb8cfe59b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:46.917283 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642a17c7-58e9-4b76-bdf6-70bb8cfe59b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:46.917562 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642a17c7-58e9-4b76-bdf6-70bb8cfe59b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:46.917831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642a17c7-58e9-4b76-bdf6-70bb8cfe59b9 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:46.918508 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642a17c7-58e9-4b76-bdf6-70bb8cfe59b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:46.934143 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642a17c7-58e9-4b76-bdf6-70bb8cfe59b9 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:46.954614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:46.969461 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1ddddb-b1c7-47d3-b7a6-823726af913c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:46.973651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0605f1-6dcc-4a3b-bdf8-4b3e0a493182 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:46.974785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0605f1-6dcc-4a3b-bdf8-4b3e0a493182 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:46.975349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0605f1-6dcc-4a3b-bdf8-4b3e0a493182 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:46.975871 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0605f1-6dcc-4a3b-bdf8-4b3e0a493182 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:46.978623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-be0605f1-6dcc-4a3b-bdf8-4b3e0a493182 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:47.083365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0605f1-6dcc-4a3b-bdf8-4b3e0a493182 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:47.087213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0605f1-6dcc-4a3b-bdf8-4b3e0a493182 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:47.100194 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 872/1768] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:46 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:48:47.121831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 897/1769] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:48:46 2022] POST /identity/v3/users => generated 289 bytes in 428 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:48:47.129464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0605f1-6dcc-4a3b-bdf8-4b3e0a493182 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:47.139148 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0605f1-6dcc-4a3b-bdf8-4b3e0a493182 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:47.145283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2517a161-e45f-466b-8fd8-312731344cfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:47.145975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2517a161-e45f-466b-8fd8-312731344cfa admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:47.147250 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2517a161-e45f-466b-8fd8-312731344cfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:47.147421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2517a161-e45f-466b-8fd8-312731344cfa admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:47.148850 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2517a161-e45f-466b-8fd8-312731344cfa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:47.150767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2517a161-e45f-466b-8fd8-312731344cfa admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:47.168253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 873/1770] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:47 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:47.172685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642a17c7-58e9-4b76-bdf6-70bb8cfe59b9 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:47.188993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642a17c7-58e9-4b76-bdf6-70bb8cfe59b9 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:47.209883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a3038f-c6d6-4633-97b3-261936ea7dea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:47.210282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a3038f-c6d6-4633-97b3-261936ea7dea admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:47.210570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a3038f-c6d6-4633-97b3-261936ea7dea admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:47.210901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a3038f-c6d6-4633-97b3-261936ea7dea admin admin] PATH_INFO: `/v3/projects/793420cc93aa45d1a072fa9a79d50cd3/users/caffa5a85feb4467b2b1c5a146f140ef/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:47.250517 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a3038f-c6d6-4633-97b3-261936ea7dea admin admin] RBAC: Authorizing `identity:create_grant(project_id=793420cc93aa45d1a072fa9a79d50cd3, user_id=caffa5a85feb4467b2b1c5a146f140ef, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:47.254123 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a3038f-c6d6-4633-97b3-261936ea7dea admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:47.360931 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 898/1771] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:47 2022] PUT /identity/v3/projects/793420cc93aa45d1a072fa9a79d50cd3/users/caffa5a85feb4467b2b1c5a146f140ef/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:47.365277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2517a161-e45f-466b-8fd8-312731344cfa admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:47.385270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2517a161-e45f-466b-8fd8-312731344cfa admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:47.589400 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0fecc7-45f9-431c-aeee-36900173d22b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:47.590028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0fecc7-45f9-431c-aeee-36900173d22b admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:47.590515 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0fecc7-45f9-431c-aeee-36900173d22b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:47.591220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0fecc7-45f9-431c-aeee-36900173d22b admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:47.591882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0fecc7-45f9-431c-aeee-36900173d22b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:47.593487 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0fecc7-45f9-431c-aeee-36900173d22b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:47.610274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 874/1772] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:48:47 2022] GET /identity/v3/roles => generated 1420 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:48:47.610567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a3038f-c6d6-4633-97b3-261936ea7dea admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:47.632026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a3038f-c6d6-4633-97b3-261936ea7dea admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:47.652728 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c84d9e9-cf3a-4563-abbe-f4d44957c1e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:47.653559 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c84d9e9-cf3a-4563-abbe-f4d44957c1e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:47.653700 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c84d9e9-cf3a-4563-abbe-f4d44957c1e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:47.653983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c84d9e9-cf3a-4563-abbe-f4d44957c1e1 admin admin] PATH_INFO: `/v3/projects/793420cc93aa45d1a072fa9a79d50cd3/users/caffa5a85feb4467b2b1c5a146f140ef/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:47.665304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c84d9e9-cf3a-4563-abbe-f4d44957c1e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=793420cc93aa45d1a072fa9a79d50cd3, user_id=caffa5a85feb4467b2b1c5a146f140ef, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:47.668372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c84d9e9-cf3a-4563-abbe-f4d44957c1e1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:47.704977 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 899/1773] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:48:47 2022] PUT /identity/v3/projects/793420cc93aa45d1a072fa9a79d50cd3/users/caffa5a85feb4467b2b1c5a146f140ef/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:48:47.715829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013a4e12-8040-4fa7-9e55-7498f8cafe48 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:47.716323 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013a4e12-8040-4fa7-9e55-7498f8cafe48 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:47.716744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013a4e12-8040-4fa7-9e55-7498f8cafe48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:47.812017 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-013a4e12-8040-4fa7-9e55-7498f8cafe48 None None] MFA Rules not processed for user `caffa5a85feb4467b2b1c5a146f140ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:47.813796 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-013a4e12-8040-4fa7-9e55-7498f8cafe48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:47.948701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:48.080294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 875/1774] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:47 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 365 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:48.291639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c84d9e9-cf3a-4563-abbe-f4d44957c1e1 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:48.454085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c84d9e9-cf3a-4563-abbe-f4d44957c1e1 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:48.471345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8485807f-1b39-4aab-83b4-e28f2fc2ff86 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:48.472209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8485807f-1b39-4aab-83b4-e28f2fc2ff86 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:48.472649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8485807f-1b39-4aab-83b4-e28f2fc2ff86 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:48.473090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8485807f-1b39-4aab-83b4-e28f2fc2ff86 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:48.501257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8485807f-1b39-4aab-83b4-e28f2fc2ff86 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:48.502796 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8485807f-1b39-4aab-83b4-e28f2fc2ff86 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:48.517626 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 900/1775] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:48 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:48:48.700779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203bf00d-3c7e-4128-aa9b-58b6fe4a0c32 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:48.701370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203bf00d-3c7e-4128-aa9b-58b6fe4a0c32 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:48.701370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203bf00d-3c7e-4128-aa9b-58b6fe4a0c32 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:48.703601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 876/1776] 10.222.0.65 () {54 vars in 921 bytes} [Wed Aug 24 03:48:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:48:48.755872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72edee65-d80c-48a7-8e91-2f4e4cdf9184 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:48.757057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72edee65-d80c-48a7-8e91-2f4e4cdf9184 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:48.757171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72edee65-d80c-48a7-8e91-2f4e4cdf9184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:48.847720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-72edee65-d80c-48a7-8e91-2f4e4cdf9184 None None] MFA Rules not processed for user `e5896018712d4b82a781365e478ee855`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:48.853162 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-72edee65-d80c-48a7-8e91-2f4e4cdf9184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:49.033308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203bf00d-3c7e-4128-aa9b-58b6fe4a0c32 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:49.063110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:49.067893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 901/1777] 10.222.0.65 () {58 vars in 1018 bytes} [Wed Aug 24 03:48:48 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 313 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:48:49.072722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203bf00d-3c7e-4128-aa9b-58b6fe4a0c32 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:49.082730 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4abdd5-b611-494f-a8ff-81f264f008e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:49.083601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4abdd5-b611-494f-a8ff-81f264f008e6 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:49.085108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4abdd5-b611-494f-a8ff-81f264f008e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:49.085473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4abdd5-b611-494f-a8ff-81f264f008e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:49.289102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4abdd5-b611-494f-a8ff-81f264f008e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:49.290877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4abdd5-b611-494f-a8ff-81f264f008e6 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:49.319935 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 877/1778] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:49 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:48:49.569984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72edee65-d80c-48a7-8e91-2f4e4cdf9184 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:49.571612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-72edee65-d80c-48a7-8e91-2f4e4cdf9184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:49.586709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72edee65-d80c-48a7-8e91-2f4e4cdf9184 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:49.601423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530e9715-86fb-427e-8a96-ada848848cfb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:49.602064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530e9715-86fb-427e-8a96-ada848848cfb service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:49.602367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530e9715-86fb-427e-8a96-ada848848cfb service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:49.602640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530e9715-86fb-427e-8a96-ada848848cfb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:49.621563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530e9715-86fb-427e-8a96-ada848848cfb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:49.623430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530e9715-86fb-427e-8a96-ada848848cfb service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:49.651401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 902/1779] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:50.961347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6bb3c5-72bd-47bf-be0b-31f2f4c4eda1 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:50.963024 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6bb3c5-72bd-47bf-be0b-31f2f4c4eda1 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:50.963550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6bb3c5-72bd-47bf-be0b-31f2f4c4eda1 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:50.965670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 878/1780] 10.222.0.65 () {54 vars in 921 bytes} [Wed Aug 24 03:48:50 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:48:50.995243 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4336e3-dd5e-4976-8a00-83707c5cdb27 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:50.996327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4336e3-dd5e-4976-8a00-83707c5cdb27 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:50.996840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4336e3-dd5e-4976-8a00-83707c5cdb27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:51.076809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-9a4336e3-dd5e-4976-8a00-83707c5cdb27 None None] MFA Rules not processed for user `e5896018712d4b82a781365e478ee855`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:51.083321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9a4336e3-dd5e-4976-8a00-83707c5cdb27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:51.125976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 903/1781] 10.222.0.65 () {58 vars in 1018 bytes} [Wed Aug 24 03:48:50 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:51.280963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6bb3c5-72bd-47bf-be0b-31f2f4c4eda1 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:51.283306 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-2e6bb3c5-72bd-47bf-be0b-31f2f4c4eda1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:51.314475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6bb3c5-72bd-47bf-be0b-31f2f4c4eda1 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:51.332419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1daaaf-d3aa-4b60-9847-b65807afd3dc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:51.334553 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1daaaf-d3aa-4b60-9847-b65807afd3dc service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:51.335279 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1daaaf-d3aa-4b60-9847-b65807afd3dc service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:51.335933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1daaaf-d3aa-4b60-9847-b65807afd3dc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:51.354103 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1daaaf-d3aa-4b60-9847-b65807afd3dc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:51.357319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1daaaf-d3aa-4b60-9847-b65807afd3dc service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:51.395660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 879/1782] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:52.555276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:52.736456 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:53.845323 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:54.312201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:54.952321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:56.066855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:57.285399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:48:57.310452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4336e3-dd5e-4976-8a00-83707c5cdb27 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:57.429849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4336e3-dd5e-4976-8a00-83707c5cdb27 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:57.429849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:57.429849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:57.429849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:57.430133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] PATH_INFO: `/v3/users/a7e2e869af8143339399b2e5e9d29548` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:57.430956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] RBAC: Authorizing `identity:delete_user(user_id=a7e2e869af8143339399b2e5e9d29548)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:57.433446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:57.744157 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a7e2e869af8143339399b2e5e9d29548'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:57.746907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a7e2e869af8143339399b2e5e9d29548'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:57.771673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a7e2e869af8143339399b2e5e9d29548'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:57.772761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a7e2e869af8143339399b2e5e9d29548'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:57.890745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a7e2e869af8143339399b2e5e9d29548'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:48:57.919467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591551db-31df-4b01-a15f-592e8609a9c4 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:57.919661 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 904/1783] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:48:57 2022] DELETE /identity/v3/users/a7e2e869af8143339399b2e5e9d29548 => generated 0 bytes in 605 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:48:57.919738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591551db-31df-4b01-a15f-592e8609a9c4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:57.920003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591551db-31df-4b01-a15f-592e8609a9c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:57.948801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-591551db-31df-4b01-a15f-592e8609a9c4 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:48:57.950022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-591551db-31df-4b01-a15f-592e8609a9c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:58.071950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 880/1784] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:48:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:48:58.080235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:48:58.082433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:48:58.161117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b07dd24-c508-446a-96c8-66e38087490e admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:48:58.167820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9316ac-3648-4964-97e2-5413b22e6f41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:48:58.177204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9316ac-3648-4964-97e2-5413b22e6f41 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:48:58.178381 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9316ac-3648-4964-97e2-5413b22e6f41 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:48:58.182422 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9316ac-3648-4964-97e2-5413b22e6f41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:48:58.196851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9316ac-3648-4964-97e2-5413b22e6f41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:48:58.198790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9316ac-3648-4964-97e2-5413b22e6f41 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:48:58.211778 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 905/1785] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:48:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:48:59.142018 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:00.035591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591551db-31df-4b01-a15f-592e8609a9c4 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:00.038300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-591551db-31df-4b01-a15f-592e8609a9c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:00.072351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591551db-31df-4b01-a15f-592e8609a9c4 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:00.081878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:00.088179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:00.088179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:00.088179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] PATH_INFO: `/v3/projects/c1e6159c2bba4176913d8775ae1f227e` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:00.088179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1e6159c2bba4176913d8775ae1f227e)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:00.088655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:00.271562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1e6159c2bba4176913d8775ae1f227e', 'request_id': 'req-ba38e651-028a-4788-b816-ff398e938fa6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:00.287167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1e6159c2bba4176913d8775ae1f227e', 'request_id': 'req-ba38e651-028a-4788-b816-ff398e938fa6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:00.300141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1e6159c2bba4176913d8775ae1f227e', 'request_id': 'req-ba38e651-028a-4788-b816-ff398e938fa6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:00.515373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] The token cache is being invalidate because project c1e6159c2bba4176913d8775ae1f227e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:49:00.527973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:00.529108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:00.533947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 881/1786] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:49:00 2022] DELETE /identity/v3/projects/c1e6159c2bba4176913d8775ae1f227e => generated 0 bytes in 496 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:00.534918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9316ac-3648-4964-97e2-5413b22e6f41 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:00.548096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9316ac-3648-4964-97e2-5413b22e6f41 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:00.700534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:00.718645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:00.718645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:00.718645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] PATH_INFO: `/v3/users/71b666a069b4402e96a6898d76a12f94` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:00.718645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] RBAC: Authorizing `identity:delete_user(user_id=71b666a069b4402e96a6898d76a12f94)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:00.718645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:00.960529 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:01.093499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71b666a069b4402e96a6898d76a12f94'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:01.094219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71b666a069b4402e96a6898d76a12f94'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:01.102203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71b666a069b4402e96a6898d76a12f94'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:01.102981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71b666a069b4402e96a6898d76a12f94'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:01.182312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71b666a069b4402e96a6898d76a12f94'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:01.200837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 906/1787] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:49:00 2022] DELETE /identity/v3/users/71b666a069b4402e96a6898d76a12f94 => generated 0 bytes in 667 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:01.807270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:01.809747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:01.840156 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba38e651-028a-4788-b816-ff398e938fa6 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:01.965698 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d850f05-c088-46c9-a18d-9e79b68671c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:01.968205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d850f05-c088-46c9-a18d-9e79b68671c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:01.973087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d850f05-c088-46c9-a18d-9e79b68671c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:01.975383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d850f05-c088-46c9-a18d-9e79b68671c4 admin admin] PATH_INFO: `/v3/projects/bdf427d8d68f47098907e9b7f92669cc` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:01.981210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d850f05-c088-46c9-a18d-9e79b68671c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bdf427d8d68f47098907e9b7f92669cc)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:01.985757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d850f05-c088-46c9-a18d-9e79b68671c4 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:02.281985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1d850f05-c088-46c9-a18d-9e79b68671c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bdf427d8d68f47098907e9b7f92669cc', 'request_id': 'req-1d850f05-c088-46c9-a18d-9e79b68671c4'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:02.289068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:02.292228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:02.295960 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1d850f05-c088-46c9-a18d-9e79b68671c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bdf427d8d68f47098907e9b7f92669cc', 'request_id': 'req-1d850f05-c088-46c9-a18d-9e79b68671c4'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:02.305668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1d850f05-c088-46c9-a18d-9e79b68671c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bdf427d8d68f47098907e9b7f92669cc', 'request_id': 'req-1d850f05-c088-46c9-a18d-9e79b68671c4'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:02.325510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:02.325510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696fafbe-9a14-4b23-8b11-bfc7f2a40094 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:02.398841 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3eee12-331f-4c4e-865f-d5d160fbea75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:02.399819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3eee12-331f-4c4e-865f-d5d160fbea75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:02.400315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3eee12-331f-4c4e-865f-d5d160fbea75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:02.400778 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3eee12-331f-4c4e-865f-d5d160fbea75 admin admin] PATH_INFO: `/v3/users/792280f0e991489d8273fad8d01f1fee` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:02.402222 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3eee12-331f-4c4e-865f-d5d160fbea75 admin admin] RBAC: Authorizing `identity:delete_user(user_id=792280f0e991489d8273fad8d01f1fee)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:02.404536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3eee12-331f-4c4e-865f-d5d160fbea75 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:02.476556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1d850f05-c088-46c9-a18d-9e79b68671c4 admin admin] The token cache is being invalidate because project bdf427d8d68f47098907e9b7f92669cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:49:02.477031 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1d850f05-c088-46c9-a18d-9e79b68671c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:02.477941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1d850f05-c088-46c9-a18d-9e79b68671c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:02.480702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 882/1788] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:49:01 2022] DELETE /identity/v3/projects/bdf427d8d68f47098907e9b7f92669cc => generated 0 bytes in 674 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:02.487228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0905c078-ac28-491f-b56d-3fe6d0d9555f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:02.488730 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0905c078-ac28-491f-b56d-3fe6d0d9555f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:02.489248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0905c078-ac28-491f-b56d-3fe6d0d9555f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:02.516651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-0905c078-ac28-491f-b56d-3fe6d0d9555f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:02.517886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0905c078-ac28-491f-b56d-3fe6d0d9555f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:02.529025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 883/1789] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:02.538325 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13942794-fb91-4f13-9e3a-4be7d8e05179 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:02.538604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13942794-fb91-4f13-9e3a-4be7d8e05179 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:02.539037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13942794-fb91-4f13-9e3a-4be7d8e05179 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:02.560879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-13942794-fb91-4f13-9e3a-4be7d8e05179 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:02.562105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-13942794-fb91-4f13-9e3a-4be7d8e05179 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:02.574990 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 884/1790] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:02.576117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13942794-fb91-4f13-9e3a-4be7d8e05179 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:02.588696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13942794-fb91-4f13-9e3a-4be7d8e05179 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:02.594760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194e010f-5b06-40bc-9fe6-1909974c1283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:02.596267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194e010f-5b06-40bc-9fe6-1909974c1283 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:02.596546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194e010f-5b06-40bc-9fe6-1909974c1283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:02.596849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194e010f-5b06-40bc-9fe6-1909974c1283 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:02.597534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194e010f-5b06-40bc-9fe6-1909974c1283 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:49:02.597831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194e010f-5b06-40bc-9fe6-1909974c1283 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:02.600606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194e010f-5b06-40bc-9fe6-1909974c1283 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:02.618447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 885/1791] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:49:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:49:02.622724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194e010f-5b06-40bc-9fe6-1909974c1283 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:02.635765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194e010f-5b06-40bc-9fe6-1909974c1283 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:02.641373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf1c955-2fa7-4076-bdc4-6c935b2a3a9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:02.641881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf1c955-2fa7-4076-bdc4-6c935b2a3a9b admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:02.642217 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf1c955-2fa7-4076-bdc4-6c935b2a3a9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:02.642468 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf1c955-2fa7-4076-bdc4-6c935b2a3a9b admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:02.643549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf1c955-2fa7-4076-bdc4-6c935b2a3a9b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:02.645377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf1c955-2fa7-4076-bdc4-6c935b2a3a9b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:02.723639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2b3eee12-331f-4c4e-865f-d5d160fbea75 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '792280f0e991489d8273fad8d01f1fee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:02.725062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2b3eee12-331f-4c4e-865f-d5d160fbea75 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '792280f0e991489d8273fad8d01f1fee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:02.737965 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2b3eee12-331f-4c4e-865f-d5d160fbea75 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '792280f0e991489d8273fad8d01f1fee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:02.739328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2b3eee12-331f-4c4e-865f-d5d160fbea75 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '792280f0e991489d8273fad8d01f1fee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:02.753918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:02.796259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 886/1792] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:49:02 2022] POST /identity/v3/projects => generated 387 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:49:02.797736 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf1c955-2fa7-4076-bdc4-6c935b2a3a9b admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:02.799942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ddf1c955-2fa7-4076-bdc4-6c935b2a3a9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:02.896487 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf1c955-2fa7-4076-bdc4-6c935b2a3a9b admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:02.902759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996ff4e1-b2bb-4f45-af05-5f6b8912dba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:02.903457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996ff4e1-b2bb-4f45-af05-5f6b8912dba4 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:02.903936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996ff4e1-b2bb-4f45-af05-5f6b8912dba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:02.904217 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996ff4e1-b2bb-4f45-af05-5f6b8912dba4 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:02.904881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996ff4e1-b2bb-4f45-af05-5f6b8912dba4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:02.907745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2b3eee12-331f-4c4e-865f-d5d160fbea75 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '792280f0e991489d8273fad8d01f1fee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:02.914861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996ff4e1-b2bb-4f45-af05-5f6b8912dba4 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:02.935178 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 907/1793] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:49:02 2022] DELETE /identity/v3/users/792280f0e991489d8273fad8d01f1fee => generated 0 bytes in 646 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:02.940608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d86d4f-5509-4eef-b8f3-03d443ad3e44 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:02.945273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d86d4f-5509-4eef-b8f3-03d443ad3e44 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:02.945273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d86d4f-5509-4eef-b8f3-03d443ad3e44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:02.964122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-67d86d4f-5509-4eef-b8f3-03d443ad3e44 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:02.965868 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-67d86d4f-5509-4eef-b8f3-03d443ad3e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:03.040585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 887/1794] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:49:02 2022] POST /identity/v3/users => generated 294 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:03.045066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996ff4e1-b2bb-4f45-af05-5f6b8912dba4 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:03.058846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 908/1795] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:03.059990 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996ff4e1-b2bb-4f45-af05-5f6b8912dba4 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:03.066972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cded10b6-0eda-4123-8f80-074294aba2e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:03.068250 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cded10b6-0eda-4123-8f80-074294aba2e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:03.068617 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cded10b6-0eda-4123-8f80-074294aba2e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:03.068960 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cded10b6-0eda-4123-8f80-074294aba2e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:03.069712 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cded10b6-0eda-4123-8f80-074294aba2e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:03.071522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cded10b6-0eda-4123-8f80-074294aba2e4 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:03.085564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 888/1796] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:03 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:49:03.092215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d86d4f-5509-4eef-b8f3-03d443ad3e44 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:03.097165 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-67d86d4f-5509-4eef-b8f3-03d443ad3e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:03.105371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cded10b6-0eda-4123-8f80-074294aba2e4 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:03.135797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cded10b6-0eda-4123-8f80-074294aba2e4 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:03.150189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0eebb7-f4d1-46f3-9d9a-9009907d2ff5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:03.151218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0eebb7-f4d1-46f3-9d9a-9009907d2ff5 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:03.151218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0eebb7-f4d1-46f3-9d9a-9009907d2ff5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:03.151388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0eebb7-f4d1-46f3-9d9a-9009907d2ff5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:03.154214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0eebb7-f4d1-46f3-9d9a-9009907d2ff5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:03.158601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0eebb7-f4d1-46f3-9d9a-9009907d2ff5 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:03.189599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 889/1797] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:03 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:49:03.194382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0eebb7-f4d1-46f3-9d9a-9009907d2ff5 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:03.210393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0eebb7-f4d1-46f3-9d9a-9009907d2ff5 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:03.216792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609bcf83-766d-444a-a8ef-ac5bec36d2e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:03.217530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609bcf83-766d-444a-a8ef-ac5bec36d2e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:03.217813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609bcf83-766d-444a-a8ef-ac5bec36d2e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:03.218975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609bcf83-766d-444a-a8ef-ac5bec36d2e7 admin admin] PATH_INFO: `/v3/projects/19d9dc642f58490289d27f16ec7f649e/users/833044bcafe441b7b7b004ba9a156a4f/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:03.226482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d86d4f-5509-4eef-b8f3-03d443ad3e44 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:03.231628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ac1ddb-30cc-4e68-b748-e969920d7c6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:03.232437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac1ddb-30cc-4e68-b748-e969920d7c6b service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:03.232825 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac1ddb-30cc-4e68-b748-e969920d7c6b service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:03.233241 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac1ddb-30cc-4e68-b748-e969920d7c6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:03.247985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609bcf83-766d-444a-a8ef-ac5bec36d2e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19d9dc642f58490289d27f16ec7f649e, user_id=833044bcafe441b7b7b004ba9a156a4f, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:03.249438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ac1ddb-30cc-4e68-b748-e969920d7c6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:03.251346 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ac1ddb-30cc-4e68-b748-e969920d7c6b service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:03.257259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609bcf83-766d-444a-a8ef-ac5bec36d2e7 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:03.266116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 909/1798] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:49:03.390477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 890/1799] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:03 2022] PUT /identity/v3/projects/19d9dc642f58490289d27f16ec7f649e/users/833044bcafe441b7b7b004ba9a156a4f/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:49:03.403681 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf0b314-dac0-4a87-974e-7fef7cc9157b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:03.412944 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf0b314-dac0-4a87-974e-7fef7cc9157b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:03.413494 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf0b314-dac0-4a87-974e-7fef7cc9157b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:03.470463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-4cf0b314-dac0-4a87-974e-7fef7cc9157b None None] MFA Rules not processed for user `833044bcafe441b7b7b004ba9a156a4f`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:03.472105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4cf0b314-dac0-4a87-974e-7fef7cc9157b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:03.555941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 910/1800] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:03 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:03.561188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609bcf83-766d-444a-a8ef-ac5bec36d2e7 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:03.682184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609bcf83-766d-444a-a8ef-ac5bec36d2e7 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:03.689029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac424583-5947-47ef-8568-fac40494e355 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:03.689929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac424583-5947-47ef-8568-fac40494e355 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:03.690177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac424583-5947-47ef-8568-fac40494e355 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:03.690515 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac424583-5947-47ef-8568-fac40494e355 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:03.691124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac424583-5947-47ef-8568-fac40494e355 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:03.693816 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac424583-5947-47ef-8568-fac40494e355 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:04.477126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 891/1801] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:49:03 2022] POST /identity/v3/projects => generated 387 bytes in 916 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:04.531297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf0b314-dac0-4a87-974e-7fef7cc9157b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:04.547400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf0b314-dac0-4a87-974e-7fef7cc9157b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:04.640500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd3122a-6750-4f4d-af73-ac399e4b64e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:04.641857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd3122a-6750-4f4d-af73-ac399e4b64e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:04.642367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd3122a-6750-4f4d-af73-ac399e4b64e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:04.642925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd3122a-6750-4f4d-af73-ac399e4b64e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:04.643969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd3122a-6750-4f4d-af73-ac399e4b64e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:04.646667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd3122a-6750-4f4d-af73-ac399e4b64e1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:05.002292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac424583-5947-47ef-8568-fac40494e355 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:05.004448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ac424583-5947-47ef-8568-fac40494e355 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:05.018273 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac424583-5947-47ef-8568-fac40494e355 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:05.026334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:05.026334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:05.026334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:05.026700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] PATH_INFO: `/v3/projects/20293ee4d4fe4015b2c040955e474d1a` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:05.028308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=20293ee4d4fe4015b2c040955e474d1a)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:05.029940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 911/1802] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:49:04 2022] POST /identity/v3/users => generated 300 bytes in 499 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:05.030770 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:05.033747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd3122a-6750-4f4d-af73-ac399e4b64e1 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:05.043612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd3122a-6750-4f4d-af73-ac399e4b64e1 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:05.048360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe275f19-375e-41c1-8767-154e6edd4eb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:05.055148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe275f19-375e-41c1-8767-154e6edd4eb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:05.055148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe275f19-375e-41c1-8767-154e6edd4eb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:05.055148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe275f19-375e-41c1-8767-154e6edd4eb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:05.055148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe275f19-375e-41c1-8767-154e6edd4eb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:05.055148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe275f19-375e-41c1-8767-154e6edd4eb7 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:05.064364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 912/1803] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:49:05.069336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe275f19-375e-41c1-8767-154e6edd4eb7 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:05.078852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe275f19-375e-41c1-8767-154e6edd4eb7 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:05.084172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9106c7b3-4d97-438e-b8bc-15f0c6148d79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:05.084810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9106c7b3-4d97-438e-b8bc-15f0c6148d79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:05.085087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9106c7b3-4d97-438e-b8bc-15f0c6148d79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:05.085364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9106c7b3-4d97-438e-b8bc-15f0c6148d79 admin admin] PATH_INFO: `/v3/projects/4dde09af23394eedb55dc907df9ee15b/users/e1815092a93045c5b7484c79b142f0db/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:05.101143 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9106c7b3-4d97-438e-b8bc-15f0c6148d79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4dde09af23394eedb55dc907df9ee15b, user_id=e1815092a93045c5b7484c79b142f0db, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:05.104205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9106c7b3-4d97-438e-b8bc-15f0c6148d79 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:05.234395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 913/1804] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:05 2022] PUT /identity/v3/projects/4dde09af23394eedb55dc907df9ee15b/users/e1815092a93045c5b7484c79b142f0db/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:49:05.238383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9106c7b3-4d97-438e-b8bc-15f0c6148d79 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:05.242742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '20293ee4d4fe4015b2c040955e474d1a', 'request_id': 'req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:05.248401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '20293ee4d4fe4015b2c040955e474d1a', 'request_id': 'req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:05.255431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '20293ee4d4fe4015b2c040955e474d1a', 'request_id': 'req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:05.300339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9106c7b3-4d97-438e-b8bc-15f0c6148d79 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:05.390981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] The token cache is being invalidate because project 20293ee4d4fe4015b2c040955e474d1a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:49:05.392023 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:05.394630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:05.394630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 892/1805] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:49:05 2022] DELETE /identity/v3/projects/20293ee4d4fe4015b2c040955e474d1a => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:49:05.395289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f42b9d1-3ae1-4c26-8b33-cbcc36d3404b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:05.396489 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f42b9d1-3ae1-4c26-8b33-cbcc36d3404b admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:05.396894 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f42b9d1-3ae1-4c26-8b33-cbcc36d3404b admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:05.397041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f42b9d1-3ae1-4c26-8b33-cbcc36d3404b admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:05.397676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f42b9d1-3ae1-4c26-8b33-cbcc36d3404b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:05.399253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:05.399373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f42b9d1-3ae1-4c26-8b33-cbcc36d3404b admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:05.400897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:05.410246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 914/1806] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:05 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:49:05.415873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f42b9d1-3ae1-4c26-8b33-cbcc36d3404b admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:05.417778 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1f42b9d1-3ae1-4c26-8b33-cbcc36d3404b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:05.429078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dce0ea-65a5-4651-98ec-d27cc3fe29bb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:05.431257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f42b9d1-3ae1-4c26-8b33-cbcc36d3404b admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:05.435079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:05.435767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:05.436052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:05.436263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9846f47-ec82-454e-8ce2-221f0d7f59ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:05.436400 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] PATH_INFO: `/v3/users/caffa5a85feb4467b2b1c5a146f140ef` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:05.436934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9846f47-ec82-454e-8ce2-221f0d7f59ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:05.437250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9846f47-ec82-454e-8ce2-221f0d7f59ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:05.437567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9846f47-ec82-454e-8ce2-221f0d7f59ad admin admin] PATH_INFO: `/v3/projects/4dde09af23394eedb55dc907df9ee15b/users/e1815092a93045c5b7484c79b142f0db/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:05.437720 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] RBAC: Authorizing `identity:delete_user(user_id=caffa5a85feb4467b2b1c5a146f140ef)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:05.439684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:05.440568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9846f47-ec82-454e-8ce2-221f0d7f59ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=4dde09af23394eedb55dc907df9ee15b, user_id=e1815092a93045c5b7484c79b142f0db, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:05.443486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9846f47-ec82-454e-8ce2-221f0d7f59ad admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:05.472030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 915/1807] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:05 2022] PUT /identity/v3/projects/4dde09af23394eedb55dc907df9ee15b/users/e1815092a93045c5b7484c79b142f0db/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:05.473597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540ee441-de4c-40cf-920a-c5f2c0766f20 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:05.474777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540ee441-de4c-40cf-920a-c5f2c0766f20 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:05.474777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540ee441-de4c-40cf-920a-c5f2c0766f20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:05.513229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-540ee441-de4c-40cf-920a-c5f2c0766f20 None None] MFA Rules not processed for user `e1815092a93045c5b7484c79b142f0db`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:05.514557 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-540ee441-de4c-40cf-920a-c5f2c0766f20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:05.614529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 916/1808] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:05 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:05.639310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540ee441-de4c-40cf-920a-c5f2c0766f20 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:05.640757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-540ee441-de4c-40cf-920a-c5f2c0766f20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:05.714247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540ee441-de4c-40cf-920a-c5f2c0766f20 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:05.717401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d11e1c-01a1-4290-afa1-e3807460c316 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:05.718074 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d11e1c-01a1-4290-afa1-e3807460c316 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:05.718399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d11e1c-01a1-4290-afa1-e3807460c316 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:05.718724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d11e1c-01a1-4290-afa1-e3807460c316 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:05.724886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'caffa5a85feb4467b2b1c5a146f140ef'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:05.726016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'caffa5a85feb4467b2b1c5a146f140ef'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:05.729310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d11e1c-01a1-4290-afa1-e3807460c316 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:05.730692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d11e1c-01a1-4290-afa1-e3807460c316 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:05.733848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'caffa5a85feb4467b2b1c5a146f140ef'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:05.742260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 917/1809] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:05 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:49:05.748865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'caffa5a85feb4467b2b1c5a146f140ef'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:05.753567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d72645d-a7e0-4c59-9821-32d8022542ab None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:05.754623 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d72645d-a7e0-4c59-9821-32d8022542ab None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:05.756234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d72645d-a7e0-4c59-9821-32d8022542ab None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:05.760324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 918/1810] 10.222.0.65 () {54 vars in 921 bytes} [Wed Aug 24 03:49:05 2022] GET /identity => generated 270 bytes in 17 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:49:05.767712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f19ef0-3111-4fd8-b116-ae53711c268a None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:05.768590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f19ef0-3111-4fd8-b116-ae53711c268a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:05.768590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f19ef0-3111-4fd8-b116-ae53711c268a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:05.792128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-19f19ef0-3111-4fd8-b116-ae53711c268a None None] MFA Rules not processed for user `e5896018712d4b82a781365e478ee855`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:05.797187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-19f19ef0-3111-4fd8-b116-ae53711c268a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:05.822896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'caffa5a85feb4467b2b1c5a146f140ef'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:05.839110 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 893/1811] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:49:05 2022] DELETE /identity/v3/users/caffa5a85feb4467b2b1c5a146f140ef => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:05.909344 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 919/1812] 10.222.0.65 () {58 vars in 1018 bytes} [Wed Aug 24 03:49:05 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:49:06.010689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:06.012298 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:06.146399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349e9061-fc1e-4f43-8bf7-9346fbe0a221 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:06.168171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f31153-6def-4b32-a8f1-559fcc7882ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:06.168860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f31153-6def-4b32-a8f1-559fcc7882ec service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:06.183666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f31153-6def-4b32-a8f1-559fcc7882ec service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:06.183666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f31153-6def-4b32-a8f1-559fcc7882ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:06.183666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f31153-6def-4b32-a8f1-559fcc7882ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:06.184932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f31153-6def-4b32-a8f1-559fcc7882ec service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:06.198044 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 894/1813] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:49:06.373171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f19ef0-3111-4fd8-b116-ae53711c268a None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:06.374731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-19f19ef0-3111-4fd8-b116-ae53711c268a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:06.494103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f19ef0-3111-4fd8-b116-ae53711c268a None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:06.509249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:06.561647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:06.562238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:06.562588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:06.562927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed admin admin] PATH_INFO: `/v3/projects/793420cc93aa45d1a072fa9a79d50cd3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:06.564107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=793420cc93aa45d1a072fa9a79d50cd3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:06.566262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:06.799685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '793420cc93aa45d1a072fa9a79d50cd3', 'request_id': 'req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:06.809039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '793420cc93aa45d1a072fa9a79d50cd3', 'request_id': 'req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:06.970244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '793420cc93aa45d1a072fa9a79d50cd3', 'request_id': 'req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:06.987957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed admin admin] The token cache is being invalidate because project 793420cc93aa45d1a072fa9a79d50cd3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:49:06.988231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:06.989042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-46cc9ca7-5b84-4593-a968-d8eefeacc6ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:06.995612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 920/1814] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:49:06 2022] DELETE /identity/v3/projects/793420cc93aa45d1a072fa9a79d50cd3 => generated 0 bytes in 620 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:06.997461 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7905d2d-9c11-4070-867a-a6de9841c592 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:06.998863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7905d2d-9c11-4070-867a-a6de9841c592 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:06.999968 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7905d2d-9c11-4070-867a-a6de9841c592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:07.026662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-b7905d2d-9c11-4070-867a-a6de9841c592 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:07.028096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b7905d2d-9c11-4070-867a-a6de9841c592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:07.138366 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 895/1815] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:49:07.147750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b0176e-a398-49ca-97db-e2c3199652d3 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:07.148135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b0176e-a398-49ca-97db-e2c3199652d3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:07.148586 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b0176e-a398-49ca-97db-e2c3199652d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:07.193025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-55b0176e-a398-49ca-97db-e2c3199652d3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:07.194287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-55b0176e-a398-49ca-97db-e2c3199652d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:07.226377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 921/1816] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:07.230952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7905d2d-9c11-4070-867a-a6de9841c592 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:07.272983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7905d2d-9c11-4070-867a-a6de9841c592 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:07.285706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9114c99-748c-4826-8c3e-64777fa6d730 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:07.286789 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9114c99-748c-4826-8c3e-64777fa6d730 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:07.286977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9114c99-748c-4826-8c3e-64777fa6d730 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:07.287072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9114c99-748c-4826-8c3e-64777fa6d730 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:07.289745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9114c99-748c-4826-8c3e-64777fa6d730 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:49:07.290054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9114c99-748c-4826-8c3e-64777fa6d730 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:07.291714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9114c99-748c-4826-8c3e-64777fa6d730 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:07.295800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b0176e-a398-49ca-97db-e2c3199652d3 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:07.297979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-55b0176e-a398-49ca-97db-e2c3199652d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:07.326737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b0176e-a398-49ca-97db-e2c3199652d3 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:07.330475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f386cc-707f-478f-80e5-40d4cacd0f40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:07.331440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 896/1817] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:49:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:49:07.331744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f386cc-707f-478f-80e5-40d4cacd0f40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:07.332201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f386cc-707f-478f-80e5-40d4cacd0f40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:07.332636 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f386cc-707f-478f-80e5-40d4cacd0f40 admin admin] PATH_INFO: `/v3/users/833044bcafe441b7b7b004ba9a156a4f` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:07.333931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9114c99-748c-4826-8c3e-64777fa6d730 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:07.335898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f386cc-707f-478f-80e5-40d4cacd0f40 admin admin] RBAC: Authorizing `identity:delete_user(user_id=833044bcafe441b7b7b004ba9a156a4f)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:07.338795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f386cc-707f-478f-80e5-40d4cacd0f40 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:07.347463 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9114c99-748c-4826-8c3e-64777fa6d730 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:07.353699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fc92c4-757d-4831-9e0b-d8454f06961b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:07.355264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fc92c4-757d-4831-9e0b-d8454f06961b admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:07.358459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fc92c4-757d-4831-9e0b-d8454f06961b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:07.358971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fc92c4-757d-4831-9e0b-d8454f06961b admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:07.360409 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fc92c4-757d-4831-9e0b-d8454f06961b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:07.363203 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fc92c4-757d-4831-9e0b-d8454f06961b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:07.540292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 897/1818] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:49:07 2022] POST /identity/v3/projects => generated 381 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:07.542196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fc92c4-757d-4831-9e0b-d8454f06961b admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:07.672443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fc92c4-757d-4831-9e0b-d8454f06961b admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:07.682339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cf5ffb-07f7-4334-9a2c-aa2270d03ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:07.683339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cf5ffb-07f7-4334-9a2c-aa2270d03ee1 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:07.683815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cf5ffb-07f7-4334-9a2c-aa2270d03ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:07.684292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cf5ffb-07f7-4334-9a2c-aa2270d03ee1 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:07.685433 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cf5ffb-07f7-4334-9a2c-aa2270d03ee1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:07.689193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cf5ffb-07f7-4334-9a2c-aa2270d03ee1 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:07.712864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a5f386cc-707f-478f-80e5-40d4cacd0f40 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '833044bcafe441b7b7b004ba9a156a4f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:07.713876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a5f386cc-707f-478f-80e5-40d4cacd0f40 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '833044bcafe441b7b7b004ba9a156a4f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:07.721604 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a5f386cc-707f-478f-80e5-40d4cacd0f40 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '833044bcafe441b7b7b004ba9a156a4f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:07.722402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a5f386cc-707f-478f-80e5-40d4cacd0f40 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '833044bcafe441b7b7b004ba9a156a4f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:07.823794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 898/1819] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:49:07 2022] POST /identity/v3/users => generated 291 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:07.834207 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cf5ffb-07f7-4334-9a2c-aa2270d03ee1 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:07.836002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-76cf5ffb-07f7-4334-9a2c-aa2270d03ee1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:07.845071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a5f386cc-707f-478f-80e5-40d4cacd0f40 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '833044bcafe441b7b7b004ba9a156a4f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:07.852046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cf5ffb-07f7-4334-9a2c-aa2270d03ee1 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:07.858805 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a60047-6c4d-423c-837c-42c5e0c69071 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:07.859448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a60047-6c4d-423c-837c-42c5e0c69071 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:07.860577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a60047-6c4d-423c-837c-42c5e0c69071 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:07.860577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a60047-6c4d-423c-837c-42c5e0c69071 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:07.861256 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a60047-6c4d-423c-837c-42c5e0c69071 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:07.863566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a60047-6c4d-423c-837c-42c5e0c69071 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:07.874294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 922/1820] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:49:07 2022] DELETE /identity/v3/users/833044bcafe441b7b7b004ba9a156a4f => generated 0 bytes in 579 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:07.878894 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fbe9c9-4275-464e-b11c-6ae7a54d862c None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:07.880885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 899/1821] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:07 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:49:07.881085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fbe9c9-4275-464e-b11c-6ae7a54d862c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:07.881085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fbe9c9-4275-464e-b11c-6ae7a54d862c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:07.884073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a60047-6c4d-423c-837c-42c5e0c69071 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:07.901583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-63fbe9c9-4275-464e-b11c-6ae7a54d862c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:07.903789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-63fbe9c9-4275-464e-b11c-6ae7a54d862c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:07.914016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a60047-6c4d-423c-837c-42c5e0c69071 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:08.009765 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 923/1822] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:49:08.021504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fbe9c9-4275-464e-b11c-6ae7a54d862c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:08.023487 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-63fbe9c9-4275-464e-b11c-6ae7a54d862c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:08.035123 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfeba49-a427-47e4-9714-e135919060b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:08.035802 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfeba49-a427-47e4-9714-e135919060b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:08.037132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfeba49-a427-47e4-9714-e135919060b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:08.037736 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfeba49-a427-47e4-9714-e135919060b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:08.038593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfeba49-a427-47e4-9714-e135919060b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:08.040397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfeba49-a427-47e4-9714-e135919060b8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:08.052115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 900/1823] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:07 2022] GET /identity/v3/roles => generated 1420 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:49:08.056759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfeba49-a427-47e4-9714-e135919060b8 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:08.069296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfeba49-a427-47e4-9714-e135919060b8 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:08.078362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a592c1da-6592-4621-aa20-3533a8ae4166 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:08.079419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a592c1da-6592-4621-aa20-3533a8ae4166 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:08.079861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a592c1da-6592-4621-aa20-3533a8ae4166 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:08.080368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a592c1da-6592-4621-aa20-3533a8ae4166 admin admin] PATH_INFO: `/v3/projects/8db3ffb2a482486c9e73cea71c7c8ada/users/3746ce7bf8b342bfacd7ee4fa542fa78/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:08.100926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a592c1da-6592-4621-aa20-3533a8ae4166 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8db3ffb2a482486c9e73cea71c7c8ada, user_id=3746ce7bf8b342bfacd7ee4fa542fa78, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:08.101210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fbe9c9-4275-464e-b11c-6ae7a54d862c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:08.103841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a592c1da-6592-4621-aa20-3533a8ae4166 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:08.105260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ec189f-c2ac-43cf-b759-82305669f3eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:08.105903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec189f-c2ac-43cf-b759-82305669f3eb service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:08.106208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec189f-c2ac-43cf-b759-82305669f3eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:08.106482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec189f-c2ac-43cf-b759-82305669f3eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:08.121090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ec189f-c2ac-43cf-b759-82305669f3eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:08.121090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ec189f-c2ac-43cf-b759-82305669f3eb service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:08.132008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 924/1824] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:49:08.215185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 901/1825] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:08 2022] PUT /identity/v3/projects/8db3ffb2a482486c9e73cea71c7c8ada/users/3746ce7bf8b342bfacd7ee4fa542fa78/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:49:08.225838 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35266fe9-6d67-47e8-ab18-3a8e91dffe54 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:08.226264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35266fe9-6d67-47e8-ab18-3a8e91dffe54 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:08.226552 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35266fe9-6d67-47e8-ab18-3a8e91dffe54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:08.272035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-35266fe9-6d67-47e8-ab18-3a8e91dffe54 None None] MFA Rules not processed for user `3746ce7bf8b342bfacd7ee4fa542fa78`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:08.280877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-35266fe9-6d67-47e8-ab18-3a8e91dffe54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:08.331895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=90,keystone:DELETE=4,keystone:INSERT=6 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:08.355778 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 925/1826] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:08 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:08.360920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a592c1da-6592-4621-aa20-3533a8ae4166 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:08.492446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a592c1da-6592-4621-aa20-3533a8ae4166 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:08.499025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaa1002-1b69-40b4-9091-a1266d39bf1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:08.499025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaa1002-1b69-40b4-9091-a1266d39bf1d admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:08.499025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaa1002-1b69-40b4-9091-a1266d39bf1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:08.499025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaa1002-1b69-40b4-9091-a1266d39bf1d admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:08.499432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaa1002-1b69-40b4-9091-a1266d39bf1d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:08.501502 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaa1002-1b69-40b4-9091-a1266d39bf1d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:08.624406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 902/1827] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:49:08 2022] POST /identity/v3/projects => generated 383 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:49:08.629093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35266fe9-6d67-47e8-ab18-3a8e91dffe54 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:08.642219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35266fe9-6d67-47e8-ab18-3a8e91dffe54 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:08.717259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b91800b-fece-42da-9c5a-4a158bdbd4b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:08.717872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b91800b-fece-42da-9c5a-4a158bdbd4b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:08.718175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b91800b-fece-42da-9c5a-4a158bdbd4b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:08.718450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b91800b-fece-42da-9c5a-4a158bdbd4b8 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:08.719179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b91800b-fece-42da-9c5a-4a158bdbd4b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:08.721107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b91800b-fece-42da-9c5a-4a158bdbd4b8 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:08.743934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaa1002-1b69-40b4-9091-a1266d39bf1d admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:08.745763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-aaaa1002-1b69-40b4-9091-a1266d39bf1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:08.759907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaa1002-1b69-40b4-9091-a1266d39bf1d admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:08.765503 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:08.766264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:08.771963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:08.777646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820 admin admin] PATH_INFO: `/v3/projects/19d9dc642f58490289d27f16ec7f649e` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:08.779197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820 admin admin] RBAC: Authorizing `identity:delete_project(project_id=19d9dc642f58490289d27f16ec7f649e)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:08.781762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:08.798237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 926/1828] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:49:08 2022] POST /identity/v3/users => generated 298 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:49:08.802237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b91800b-fece-42da-9c5a-4a158bdbd4b8 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:08.812684 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b91800b-fece-42da-9c5a-4a158bdbd4b8 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:08.820730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea167f2-184c-4313-b65d-d1886a9deaac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:08.820730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea167f2-184c-4313-b65d-d1886a9deaac admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:08.821004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea167f2-184c-4313-b65d-d1886a9deaac admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:08.821123 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea167f2-184c-4313-b65d-d1886a9deaac admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:08.822582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea167f2-184c-4313-b65d-d1886a9deaac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:08.824267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea167f2-184c-4313-b65d-d1886a9deaac admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:08.836752 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 927/1829] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:08 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:49:08.841778 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea167f2-184c-4313-b65d-d1886a9deaac admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:08.854733 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea167f2-184c-4313-b65d-d1886a9deaac admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:08.868079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b7dde4-223d-45ac-a172-71126a913ddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:08.868079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b7dde4-223d-45ac-a172-71126a913ddc admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:08.868079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b7dde4-223d-45ac-a172-71126a913ddc admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:08.868079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b7dde4-223d-45ac-a172-71126a913ddc admin admin] PATH_INFO: `/v3/projects/ec2f7127634d42ab907b269a790d48fd/users/9238f5b1fb4145fe9f131fe99e889cd5/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:08.888903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b7dde4-223d-45ac-a172-71126a913ddc admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec2f7127634d42ab907b269a790d48fd, user_id=9238f5b1fb4145fe9f131fe99e889cd5, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:08.893211 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b7dde4-223d-45ac-a172-71126a913ddc admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:09.022361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 928/1830] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:08 2022] PUT /identity/v3/projects/ec2f7127634d42ab907b269a790d48fd/users/9238f5b1fb4145fe9f131fe99e889cd5/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:49:09.022361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b7dde4-223d-45ac-a172-71126a913ddc admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:09.029220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19d9dc642f58490289d27f16ec7f649e', 'request_id': 'req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:09.048531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19d9dc642f58490289d27f16ec7f649e', 'request_id': 'req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:09.072536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19d9dc642f58490289d27f16ec7f649e', 'request_id': 'req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:09.133774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b7dde4-223d-45ac-a172-71126a913ddc admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:09.137106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1767b9-356d-4e76-86b9-f29f2c4474af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:09.137967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1767b9-356d-4e76-86b9-f29f2c4474af admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:09.138479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1767b9-356d-4e76-86b9-f29f2c4474af admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:09.138854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1767b9-356d-4e76-86b9-f29f2c4474af admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:09.139782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1767b9-356d-4e76-86b9-f29f2c4474af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:09.141655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1767b9-356d-4e76-86b9-f29f2c4474af admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:09.161601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 929/1831] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:09 2022] GET /identity/v3/roles => generated 1420 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:49:09.162954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1767b9-356d-4e76-86b9-f29f2c4474af admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:09.175289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1767b9-356d-4e76-86b9-f29f2c4474af admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:09.182062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da05d449-5a4d-4371-b79d-654de91003b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:09.182910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05d449-5a4d-4371-b79d-654de91003b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:09.183269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05d449-5a4d-4371-b79d-654de91003b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:09.183638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05d449-5a4d-4371-b79d-654de91003b3 admin admin] PATH_INFO: `/v3/projects/ec2f7127634d42ab907b269a790d48fd/users/9238f5b1fb4145fe9f131fe99e889cd5/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:09.186323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da05d449-5a4d-4371-b79d-654de91003b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec2f7127634d42ab907b269a790d48fd, user_id=9238f5b1fb4145fe9f131fe99e889cd5, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:09.189234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da05d449-5a4d-4371-b79d-654de91003b3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:09.205997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820 admin admin] The token cache is being invalidate because project 19d9dc642f58490289d27f16ec7f649e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:49:09.206926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:09.211587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-994d7cc7-41fb-4c96-86a8-7be8ad6ab820 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:09.213509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 930/1832] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:09 2022] PUT /identity/v3/projects/ec2f7127634d42ab907b269a790d48fd/users/9238f5b1fb4145fe9f131fe99e889cd5/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:09.213650 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 903/1833] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:49:08 2022] DELETE /identity/v3/projects/19d9dc642f58490289d27f16ec7f649e => generated 0 bytes in 470 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:49:09.225435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2f4cce-d95d-4c8f-912e-7710272204a6 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:09.225721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2f4cce-d95d-4c8f-912e-7710272204a6 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:09.225992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2f4cce-d95d-4c8f-912e-7710272204a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:09.226698 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da05d449-5a4d-4371-b79d-654de91003b3 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:09.229512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-da05d449-5a4d-4371-b79d-654de91003b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:09.260967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-4d2f4cce-d95d-4c8f-912e-7710272204a6 None None] MFA Rules not processed for user `9238f5b1fb4145fe9f131fe99e889cd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:09.262551 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4d2f4cce-d95d-4c8f-912e-7710272204a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:09.316918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da05d449-5a4d-4371-b79d-654de91003b3 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:09.331919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:09.334111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:09.335308 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:09.337959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] PATH_INFO: `/v3/users/e1815092a93045c5b7484c79b142f0db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:09.337959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1815092a93045c5b7484c79b142f0db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:09.340811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:09.362179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 904/1834] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:09 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:49:09.379180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2f4cce-d95d-4c8f-912e-7710272204a6 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:09.380763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4d2f4cce-d95d-4c8f-912e-7710272204a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:09.454618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2f4cce-d95d-4c8f-912e-7710272204a6 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:09.460560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18050caa-a02f-43b6-9022-a49271e1cca5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:09.461077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18050caa-a02f-43b6-9022-a49271e1cca5 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:09.461356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18050caa-a02f-43b6-9022-a49271e1cca5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:09.461632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18050caa-a02f-43b6-9022-a49271e1cca5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:09.477165 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18050caa-a02f-43b6-9022-a49271e1cca5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:09.478671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18050caa-a02f-43b6-9022-a49271e1cca5 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:09.494623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 905/1835] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:09 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:49:09.579996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1815092a93045c5b7484c79b142f0db'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:09.580796 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1815092a93045c5b7484c79b142f0db'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:09.589489 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1815092a93045c5b7484c79b142f0db'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:09.590276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1815092a93045c5b7484c79b142f0db'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:09.683044 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18050caa-a02f-43b6-9022-a49271e1cca5 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:09.702243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18050caa-a02f-43b6-9022-a49271e1cca5 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:09.706569 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd43848e-6e8b-418b-b4f8-a51509a24615 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:09.707363 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd43848e-6e8b-418b-b4f8-a51509a24615 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:09.707605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd43848e-6e8b-418b-b4f8-a51509a24615 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:09.707707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1815092a93045c5b7484c79b142f0db'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:09.708430 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd43848e-6e8b-418b-b4f8-a51509a24615 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:09.712585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-dd43848e-6e8b-418b-b4f8-a51509a24615 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:09.743734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 931/1836] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:49:09 2022] DELETE /identity/v3/users/e1815092a93045c5b7484c79b142f0db => generated 0 bytes in 520 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:09.824448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd43848e-6e8b-418b-b4f8-a51509a24615 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:09.826824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd43848e-6e8b-418b-b4f8-a51509a24615 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:09.855253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 906/1837] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:09 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:49:10.323379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:10.625159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:10.626635 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:10.711444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd43848e-6e8b-418b-b4f8-a51509a24615 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:10.714005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-dd43848e-6e8b-418b-b4f8-a51509a24615 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:10.738376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f8d02d-9416-410a-b5bd-5589c40b4e24 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:10.745399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd160ff-98d0-4c79-b9cf-d7e58719fe02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:10.746425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd160ff-98d0-4c79-b9cf-d7e58719fe02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:10.746728 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd160ff-98d0-4c79-b9cf-d7e58719fe02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:10.746900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd160ff-98d0-4c79-b9cf-d7e58719fe02 admin admin] PATH_INFO: `/v3/users/3746ce7bf8b342bfacd7ee4fa542fa78` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:10.749236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd160ff-98d0-4c79-b9cf-d7e58719fe02 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3746ce7bf8b342bfacd7ee4fa542fa78)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:10.752609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd160ff-98d0-4c79-b9cf-d7e58719fe02 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:10.779033 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd43848e-6e8b-418b-b4f8-a51509a24615 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:10.788973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:10.789634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:10.789930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:10.790526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f admin admin] PATH_INFO: `/v3/projects/4dde09af23394eedb55dc907df9ee15b` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:10.792313 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f admin admin] RBAC: Authorizing `identity:delete_project(project_id=4dde09af23394eedb55dc907df9ee15b)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:10.794374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:11.018327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4dde09af23394eedb55dc907df9ee15b', 'request_id': 'req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:11.026878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4dde09af23394eedb55dc907df9ee15b', 'request_id': 'req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:11.031843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4cd160ff-98d0-4c79-b9cf-d7e58719fe02 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3746ce7bf8b342bfacd7ee4fa542fa78'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:11.032830 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4cd160ff-98d0-4c79-b9cf-d7e58719fe02 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3746ce7bf8b342bfacd7ee4fa542fa78'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:11.037831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4dde09af23394eedb55dc907df9ee15b', 'request_id': 'req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:11.044179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4cd160ff-98d0-4c79-b9cf-d7e58719fe02 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3746ce7bf8b342bfacd7ee4fa542fa78'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:11.045111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4cd160ff-98d0-4c79-b9cf-d7e58719fe02 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3746ce7bf8b342bfacd7ee4fa542fa78'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:11.169969 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f admin admin] The token cache is being invalidate because project 4dde09af23394eedb55dc907df9ee15b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:49:11.170960 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:11.172342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-8387c5b4-3be0-49b0-a3cf-b3eb8792b35f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:11.175822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 907/1838] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:49:10 2022] DELETE /identity/v3/projects/4dde09af23394eedb55dc907df9ee15b => generated 0 bytes in 465 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:11.181001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcee6de-9425-4fe7-a50d-8b2ceb8a5f46 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:11.181287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcee6de-9425-4fe7-a50d-8b2ceb8a5f46 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:11.181560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcee6de-9425-4fe7-a50d-8b2ceb8a5f46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:11.218456 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-6bcee6de-9425-4fe7-a50d-8b2ceb8a5f46 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:11.221576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6bcee6de-9425-4fe7-a50d-8b2ceb8a5f46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:11.235715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4cd160ff-98d0-4c79-b9cf-d7e58719fe02 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3746ce7bf8b342bfacd7ee4fa542fa78'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:11.275486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 932/1839] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:49:10 2022] DELETE /identity/v3/users/3746ce7bf8b342bfacd7ee4fa542fa78 => generated 0 bytes in 651 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:11.279621 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de043a9-7bd0-4b47-98a7-ff0192d05555 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:11.280171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de043a9-7bd0-4b47-98a7-ff0192d05555 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:11.280890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de043a9-7bd0-4b47-98a7-ff0192d05555 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:11.314053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-2de043a9-7bd0-4b47-98a7-ff0192d05555 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:11.314776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=80,keystone:INSERT=3,keystone:DELETE=17 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:11.315492 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-2de043a9-7bd0-4b47-98a7-ff0192d05555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:11.366702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 908/1840] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:49:11.376076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29effd24-6a43-4131-925a-086db05aa8b9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:11.376365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29effd24-6a43-4131-925a-086db05aa8b9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:11.376652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29effd24-6a43-4131-925a-086db05aa8b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:11.445591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-29effd24-6a43-4131-925a-086db05aa8b9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:11.447571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-29effd24-6a43-4131-925a-086db05aa8b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:11.450865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 933/1841] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:49:11.459937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de043a9-7bd0-4b47-98a7-ff0192d05555 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:11.463071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-2de043a9-7bd0-4b47-98a7-ff0192d05555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:11.470204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 909/1842] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:49:11.472603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29effd24-6a43-4131-925a-086db05aa8b9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:11.490128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29effd24-6a43-4131-925a-086db05aa8b9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:11.496133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebfadaa-eb21-41cf-b2db-4995460ba5ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:11.496939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebfadaa-eb21-41cf-b2db-4995460ba5ff admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:11.497396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebfadaa-eb21-41cf-b2db-4995460ba5ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:11.497814 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebfadaa-eb21-41cf-b2db-4995460ba5ff admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:11.498685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ebfadaa-eb21-41cf-b2db-4995460ba5ff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:49:11.498989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ebfadaa-eb21-41cf-b2db-4995460ba5ff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:11.500974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ebfadaa-eb21-41cf-b2db-4995460ba5ff admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:11.516054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 910/1843] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:49:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:49:11.522766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebfadaa-eb21-41cf-b2db-4995460ba5ff admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:11.543390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebfadaa-eb21-41cf-b2db-4995460ba5ff admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:11.554483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f8a2e8-2b2a-4a24-87f3-09efeea61d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:11.555768 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f8a2e8-2b2a-4a24-87f3-09efeea61d7e admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:11.556055 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f8a2e8-2b2a-4a24-87f3-09efeea61d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:11.557437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f8a2e8-2b2a-4a24-87f3-09efeea61d7e admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:11.558137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f8a2e8-2b2a-4a24-87f3-09efeea61d7e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:11.563140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f8a2e8-2b2a-4a24-87f3-09efeea61d7e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:11.574106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de043a9-7bd0-4b47-98a7-ff0192d05555 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:11.584651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e8d42c-b4d5-4700-a1de-4dc9262ec28e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:11.586775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8d42c-b4d5-4700-a1de-4dc9262ec28e service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:11.588239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8d42c-b4d5-4700-a1de-4dc9262ec28e service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:11.588648 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8d42c-b4d5-4700-a1de-4dc9262ec28e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:11.608411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e8d42c-b4d5-4700-a1de-4dc9262ec28e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:11.609988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e8d42c-b4d5-4700-a1de-4dc9262ec28e service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:11.621301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 934/1844] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:49:11.707276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 911/1845] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:49:11 2022] POST /identity/v3/projects => generated 383 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:49:11.709830 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e8d42c-b4d5-4700-a1de-4dc9262ec28e service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:11.719944 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e8d42c-b4d5-4700-a1de-4dc9262ec28e service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:11.780341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d485b7c-7ac4-4883-bf81-9aa67268d9f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:11.780961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d485b7c-7ac4-4883-bf81-9aa67268d9f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:11.781225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d485b7c-7ac4-4883-bf81-9aa67268d9f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:11.781493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d485b7c-7ac4-4883-bf81-9aa67268d9f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:11.782111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d485b7c-7ac4-4883-bf81-9aa67268d9f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:11.784654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d485b7c-7ac4-4883-bf81-9aa67268d9f5 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:11.891862 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 935/1846] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:49:11 2022] POST /identity/v3/users => generated 292 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:49:11.896121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f8a2e8-2b2a-4a24-87f3-09efeea61d7e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:11.917650 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f8a2e8-2b2a-4a24-87f3-09efeea61d7e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:11.924455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4546190-5d4d-49fd-a817-a8edd09eb002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:11.925550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4546190-5d4d-49fd-a817-a8edd09eb002 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:11.926334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4546190-5d4d-49fd-a817-a8edd09eb002 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:11.927355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4546190-5d4d-49fd-a817-a8edd09eb002 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:11.928044 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4546190-5d4d-49fd-a817-a8edd09eb002 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:11.929907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4546190-5d4d-49fd-a817-a8edd09eb002 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:11.946754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 912/1847] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:11 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:49:11.950795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d485b7c-7ac4-4883-bf81-9aa67268d9f5 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:11.966670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d485b7c-7ac4-4883-bf81-9aa67268d9f5 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:11.968482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05f0dc7-394d-4c2e-a6ac-a0204f1dc4c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:11.969331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05f0dc7-394d-4c2e-a6ac-a0204f1dc4c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:11.969611 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05f0dc7-394d-4c2e-a6ac-a0204f1dc4c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:11.969887 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05f0dc7-394d-4c2e-a6ac-a0204f1dc4c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:11.970582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05f0dc7-394d-4c2e-a6ac-a0204f1dc4c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:11.972306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05f0dc7-394d-4c2e-a6ac-a0204f1dc4c2 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:11.983134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 936/1848] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:11 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:49:11.987609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4546190-5d4d-49fd-a817-a8edd09eb002 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:11.999301 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4546190-5d4d-49fd-a817-a8edd09eb002 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:12.004671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b226246d-d7d7-4133-bdd5-21f83d4801c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:12.006769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b226246d-d7d7-4133-bdd5-21f83d4801c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:12.006769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b226246d-d7d7-4133-bdd5-21f83d4801c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:12.006769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b226246d-d7d7-4133-bdd5-21f83d4801c3 admin admin] PATH_INFO: `/v3/projects/e1d407a742f140c98a8b06819a2a5b90/users/0d5b84d34cf34c20b3bd29788bafa7db/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:12.023306 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b226246d-d7d7-4133-bdd5-21f83d4801c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1d407a742f140c98a8b06819a2a5b90, user_id=0d5b84d34cf34c20b3bd29788bafa7db, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:12.026204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b226246d-d7d7-4133-bdd5-21f83d4801c3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:12.154913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 913/1849] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:11 2022] PUT /identity/v3/projects/e1d407a742f140c98a8b06819a2a5b90/users/0d5b84d34cf34c20b3bd29788bafa7db/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:12.167281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfd3b78-e901-4a83-996b-aab25cf7b743 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:12.167658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfd3b78-e901-4a83-996b-aab25cf7b743 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:12.168042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfd3b78-e901-4a83-996b-aab25cf7b743 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:12.219782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5dfd3b78-e901-4a83-996b-aab25cf7b743 None None] MFA Rules not processed for user `0d5b84d34cf34c20b3bd29788bafa7db`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:12.221393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5dfd3b78-e901-4a83-996b-aab25cf7b743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:12.235477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b226246d-d7d7-4133-bdd5-21f83d4801c3 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:12.239999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b226246d-d7d7-4133-bdd5-21f83d4801c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:12.328683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b226246d-d7d7-4133-bdd5-21f83d4801c3 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:12.328808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 937/1850] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:12 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:12.335939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:12.336711 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:12.337094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:12.337471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] PATH_INFO: `/v3/projects/8db3ffb2a482486c9e73cea71c7c8ada` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:12.339639 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8db3ffb2a482486c9e73cea71c7c8ada)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:12.340068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfd3b78-e901-4a83-996b-aab25cf7b743 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:12.342187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:12.347996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5dfd3b78-e901-4a83-996b-aab25cf7b743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:12.446532 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfd3b78-e901-4a83-996b-aab25cf7b743 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:12.452350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00982d05-c882-4378-af42-99c2a0a95c19 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:12.452991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00982d05-c882-4378-af42-99c2a0a95c19 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:12.453252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00982d05-c882-4378-af42-99c2a0a95c19 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:12.453519 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00982d05-c882-4378-af42-99c2a0a95c19 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:12.467908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00982d05-c882-4378-af42-99c2a0a95c19 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:12.469378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00982d05-c882-4378-af42-99c2a0a95c19 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:12.482298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 938/1851] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:12 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:49:12.508250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00982d05-c882-4378-af42-99c2a0a95c19 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:12.532029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00982d05-c882-4378-af42-99c2a0a95c19 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:12.541119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8db3ffb2a482486c9e73cea71c7c8ada', 'request_id': 'req-daa8cb81-3105-4e23-a52a-fcbf4b795464'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:12.555037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8db3ffb2a482486c9e73cea71c7c8ada', 'request_id': 'req-daa8cb81-3105-4e23-a52a-fcbf4b795464'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:12.564310 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8db3ffb2a482486c9e73cea71c7c8ada', 'request_id': 'req-daa8cb81-3105-4e23-a52a-fcbf4b795464'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:12.656503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bd361a-f725-4135-bdf4-9aef3a7da725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:12.657267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bd361a-f725-4135-bdf4-9aef3a7da725 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:12.657969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bd361a-f725-4135-bdf4-9aef3a7da725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:12.657969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bd361a-f725-4135-bdf4-9aef3a7da725 admin admin] PATH_INFO: `/v3/users/0d5b84d34cf34c20b3bd29788bafa7db` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:12.659215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bd361a-f725-4135-bdf4-9aef3a7da725 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d5b84d34cf34c20b3bd29788bafa7db)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:12.661278 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bd361a-f725-4135-bdf4-9aef3a7da725 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:12.732754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] The token cache is being invalidate because project 8db3ffb2a482486c9e73cea71c7c8ada was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:49:12.732981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:12.735075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:12.737824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 914/1852] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:49:12 2022] DELETE /identity/v3/projects/8db3ffb2a482486c9e73cea71c7c8ada => generated 0 bytes in 503 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:49:12.742912 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:12.744343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:12.758560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa8cb81-3105-4e23-a52a-fcbf4b795464 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:12.763511 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:12.764180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:12.764488 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:12.764810 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] PATH_INFO: `/v3/users/9238f5b1fb4145fe9f131fe99e889cd5` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:12.766023 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=9238f5b1fb4145fe9f131fe99e889cd5)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:12.768114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:12.944523 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:12.975919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-70bd361a-f725-4135-bdf4-9aef3a7da725 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d5b84d34cf34c20b3bd29788bafa7db'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:12.976794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-70bd361a-f725-4135-bdf4-9aef3a7da725 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d5b84d34cf34c20b3bd29788bafa7db'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:12.985651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-70bd361a-f725-4135-bdf4-9aef3a7da725 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d5b84d34cf34c20b3bd29788bafa7db'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:12.986413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-70bd361a-f725-4135-bdf4-9aef3a7da725 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d5b84d34cf34c20b3bd29788bafa7db'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:13.029740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9238f5b1fb4145fe9f131fe99e889cd5'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:13.030715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9238f5b1fb4145fe9f131fe99e889cd5'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:13.039735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9238f5b1fb4145fe9f131fe99e889cd5'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:13.052699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9238f5b1fb4145fe9f131fe99e889cd5'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:13.118850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-70bd361a-f725-4135-bdf4-9aef3a7da725 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d5b84d34cf34c20b3bd29788bafa7db'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:13.142864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 939/1853] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:49:12 2022] DELETE /identity/v3/users/0d5b84d34cf34c20b3bd29788bafa7db => generated 0 bytes in 634 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:13.146472 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf30b98-26cc-48ce-8686-e16bd31311aa None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:13.146815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf30b98-26cc-48ce-8686-e16bd31311aa None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:13.148212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf30b98-26cc-48ce-8686-e16bd31311aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:13.166500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-bbf30b98-26cc-48ce-8686-e16bd31311aa None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:13.167766 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-bbf30b98-26cc-48ce-8686-e16bd31311aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:13.181375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9238f5b1fb4145fe9f131fe99e889cd5'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:13.187851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 915/1854] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:49:12 2022] DELETE /identity/v3/users/9238f5b1fb4145fe9f131fe99e889cd5 => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:13.242391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 940/1855] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:49:13.252581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:13.254194 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:13.325732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd51fca0-e6d2-4d53-8532-0cd5220b65cc admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:13.330141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69b6937-defb-4d36-aa70-8ba116aca505 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:13.331414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69b6937-defb-4d36-aa70-8ba116aca505 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:13.331414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69b6937-defb-4d36-aa70-8ba116aca505 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:13.331414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69b6937-defb-4d36-aa70-8ba116aca505 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:13.426300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69b6937-defb-4d36-aa70-8ba116aca505 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:13.428904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69b6937-defb-4d36-aa70-8ba116aca505 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:13.442967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 916/1856] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:49:13.789495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf30b98-26cc-48ce-8686-e16bd31311aa None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:13.791820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-bbf30b98-26cc-48ce-8686-e16bd31311aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:13.806739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf30b98-26cc-48ce-8686-e16bd31311aa None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:13.812801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:13.813466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:13.813766 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:13.814059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc admin admin] PATH_INFO: `/v3/projects/ec2f7127634d42ab907b269a790d48fd` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:13.815271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec2f7127634d42ab907b269a790d48fd)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:13.817653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:13.981886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec2f7127634d42ab907b269a790d48fd', 'request_id': 'req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:13.992500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec2f7127634d42ab907b269a790d48fd', 'request_id': 'req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:14.022590 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:14.073726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec2f7127634d42ab907b269a790d48fd', 'request_id': 'req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:14.093362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc admin admin] The token cache is being invalidate because project ec2f7127634d42ab907b269a790d48fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:49:14.093840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:14.095913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ddf0d9ac-c562-4c5f-be82-7b024d951dcc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:14.098272 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 941/1857] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:49:13 2022] DELETE /identity/v3/projects/ec2f7127634d42ab907b269a790d48fd => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:14.103886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373b4cd3-746d-4d9e-9b22-e85b6ddc0cd3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:14.104406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373b4cd3-746d-4d9e-9b22-e85b6ddc0cd3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:14.105984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373b4cd3-746d-4d9e-9b22-e85b6ddc0cd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:14.150357 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-373b4cd3-746d-4d9e-9b22-e85b6ddc0cd3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:14.156296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-373b4cd3-746d-4d9e-9b22-e85b6ddc0cd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:14.226527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 917/1858] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:14.234924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b20dcb3-a741-4df5-9e57-8cc74bcd2563 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:14.235725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b20dcb3-a741-4df5-9e57-8cc74bcd2563 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:14.236235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b20dcb3-a741-4df5-9e57-8cc74bcd2563 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:14.264290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1b20dcb3-a741-4df5-9e57-8cc74bcd2563 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:14.265755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1b20dcb3-a741-4df5-9e57-8cc74bcd2563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:14.276719 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 942/1859] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:14.282203 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373b4cd3-746d-4d9e-9b22-e85b6ddc0cd3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:14.298364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373b4cd3-746d-4d9e-9b22-e85b6ddc0cd3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:14.305173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98970d4a-9952-4f96-b65d-ee8496b8e9cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:14.306118 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98970d4a-9952-4f96-b65d-ee8496b8e9cb admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:14.306437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98970d4a-9952-4f96-b65d-ee8496b8e9cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:14.306706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98970d4a-9952-4f96-b65d-ee8496b8e9cb admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:14.307360 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98970d4a-9952-4f96-b65d-ee8496b8e9cb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:49:14.307797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98970d4a-9952-4f96-b65d-ee8496b8e9cb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:14.310177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98970d4a-9952-4f96-b65d-ee8496b8e9cb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:14.324860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 918/1860] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:49:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:49:14.329367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b20dcb3-a741-4df5-9e57-8cc74bcd2563 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:14.346005 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b20dcb3-a741-4df5-9e57-8cc74bcd2563 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:14.354308 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b529603-0669-42d8-9f57-e02e4372d1e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:14.355231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b529603-0669-42d8-9f57-e02e4372d1e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:14.355512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b529603-0669-42d8-9f57-e02e4372d1e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:14.355795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b529603-0669-42d8-9f57-e02e4372d1e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:14.356628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b529603-0669-42d8-9f57-e02e4372d1e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:14.358864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b529603-0669-42d8-9f57-e02e4372d1e9 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:14.379146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98970d4a-9952-4f96-b65d-ee8496b8e9cb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:14.381453 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-98970d4a-9952-4f96-b65d-ee8496b8e9cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:14.400862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98970d4a-9952-4f96-b65d-ee8496b8e9cb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:14.410442 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b04020-e311-47a7-bda6-c1483fddaea9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:14.412092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b04020-e311-47a7-bda6-c1483fddaea9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:14.412678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b04020-e311-47a7-bda6-c1483fddaea9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:14.413193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b04020-e311-47a7-bda6-c1483fddaea9 admin admin] PATH_INFO: `/v3/projects/e1d407a742f140c98a8b06819a2a5b90` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:14.414904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b04020-e311-47a7-bda6-c1483fddaea9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1d407a742f140c98a8b06819a2a5b90)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:14.427693 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b04020-e311-47a7-bda6-c1483fddaea9 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:14.516443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 943/1861] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:49:14 2022] POST /identity/v3/projects => generated 367 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:14.529038 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b529603-0669-42d8-9f57-e02e4372d1e9 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:14.636551 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-38b04020-e311-47a7-bda6-c1483fddaea9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1d407a742f140c98a8b06819a2a5b90', 'request_id': 'req-38b04020-e311-47a7-bda6-c1483fddaea9'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:14.665718 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-38b04020-e311-47a7-bda6-c1483fddaea9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1d407a742f140c98a8b06819a2a5b90', 'request_id': 'req-38b04020-e311-47a7-bda6-c1483fddaea9'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:14.681597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-38b04020-e311-47a7-bda6-c1483fddaea9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1d407a742f140c98a8b06819a2a5b90', 'request_id': 'req-38b04020-e311-47a7-bda6-c1483fddaea9'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:14.692395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b529603-0669-42d8-9f57-e02e4372d1e9 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:14.772217 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-38b04020-e311-47a7-bda6-c1483fddaea9 admin admin] The token cache is being invalidate because project e1d407a742f140c98a8b06819a2a5b90 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:49:14.772509 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-38b04020-e311-47a7-bda6-c1483fddaea9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:14.773261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-38b04020-e311-47a7-bda6-c1483fddaea9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:14.775204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 919/1862] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:49:14 2022] DELETE /identity/v3/projects/e1d407a742f140c98a8b06819a2a5b90 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:14.779841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3b8620-628e-4c5f-88af-6d3bc131a009 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:14.780126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3b8620-628e-4c5f-88af-6d3bc131a009 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:14.780436 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3b8620-628e-4c5f-88af-6d3bc131a009 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:14.798277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e938882-5939-4f48-8f07-22d4dd35373c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:14.799729 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e938882-5939-4f48-8f07-22d4dd35373c admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:14.800076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e938882-5939-4f48-8f07-22d4dd35373c admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:14.800420 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e938882-5939-4f48-8f07-22d4dd35373c admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:14.801132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e938882-5939-4f48-8f07-22d4dd35373c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:14.804420 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e938882-5939-4f48-8f07-22d4dd35373c admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:14.804877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-2a3b8620-628e-4c5f-88af-6d3bc131a009 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:14.806287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-2a3b8620-628e-4c5f-88af-6d3bc131a009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:14.815094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 920/1863] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:49:14.824007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010de17f-f4d2-4db6-ae62-d9e8b8013aa3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:14.824289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010de17f-f4d2-4db6-ae62-d9e8b8013aa3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:14.824561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010de17f-f4d2-4db6-ae62-d9e8b8013aa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:14.851798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-010de17f-f4d2-4db6-ae62-d9e8b8013aa3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:14.853630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-010de17f-f4d2-4db6-ae62-d9e8b8013aa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:14.865094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 921/1864] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:49:14.871035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010de17f-f4d2-4db6-ae62-d9e8b8013aa3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:14.881384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010de17f-f4d2-4db6-ae62-d9e8b8013aa3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:14.887982 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21969184-a1b0-4053-ad77-4fe6220f1625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:14.888711 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21969184-a1b0-4053-ad77-4fe6220f1625 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:14.889073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21969184-a1b0-4053-ad77-4fe6220f1625 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:14.889520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21969184-a1b0-4053-ad77-4fe6220f1625 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:14.890290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21969184-a1b0-4053-ad77-4fe6220f1625 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:49:14.890667 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21969184-a1b0-4053-ad77-4fe6220f1625 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:14.892576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21969184-a1b0-4053-ad77-4fe6220f1625 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:14.909403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 922/1865] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:49:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:49:14.910934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21969184-a1b0-4053-ad77-4fe6220f1625 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:14.923539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21969184-a1b0-4053-ad77-4fe6220f1625 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:14.929947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e15d35-b22d-4251-897a-cf5786977453 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:14.930911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e15d35-b22d-4251-897a-cf5786977453 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:14.934406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e15d35-b22d-4251-897a-cf5786977453 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:14.934406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e15d35-b22d-4251-897a-cf5786977453 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:14.934406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e15d35-b22d-4251-897a-cf5786977453 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:14.934846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e15d35-b22d-4251-897a-cf5786977453 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:15.008916 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 944/1866] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:49:14 2022] POST /identity/v3/users => generated 284 bytes in 483 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:15.017319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e938882-5939-4f48-8f07-22d4dd35373c admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:15.023375 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0e938882-5939-4f48-8f07-22d4dd35373c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:15.058652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e938882-5939-4f48-8f07-22d4dd35373c admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:15.073487 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2220d72b-3aad-435e-8a39-008e35bad5f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:15.074346 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2220d72b-3aad-435e-8a39-008e35bad5f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:15.075590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2220d72b-3aad-435e-8a39-008e35bad5f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:15.076007 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2220d72b-3aad-435e-8a39-008e35bad5f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:15.076981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2220d72b-3aad-435e-8a39-008e35bad5f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:15.081525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2220d72b-3aad-435e-8a39-008e35bad5f3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:15.102231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 945/1867] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:15 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:49:15.106245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2220d72b-3aad-435e-8a39-008e35bad5f3 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:15.120954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2220d72b-3aad-435e-8a39-008e35bad5f3 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:15.127947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 923/1868] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:49:14 2022] POST /identity/v3/projects => generated 377 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:15.129601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8813d7-bf04-45e9-a999-269785761abb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:15.130976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8813d7-bf04-45e9-a999-269785761abb admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:15.131402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8813d7-bf04-45e9-a999-269785761abb admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:15.131684 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8813d7-bf04-45e9-a999-269785761abb admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:15.132435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8813d7-bf04-45e9-a999-269785761abb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:15.133620 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e15d35-b22d-4251-897a-cf5786977453 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:15.135999 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8813d7-bf04-45e9-a999-269785761abb admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:15.150949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 946/1869] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:15 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:49:15.152152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8813d7-bf04-45e9-a999-269785761abb admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:15.165446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8813d7-bf04-45e9-a999-269785761abb admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:15.209740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e15d35-b22d-4251-897a-cf5786977453 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:15.215941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9744591e-347a-4af5-b8be-a5cedc340d84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:15.217351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9744591e-347a-4af5-b8be-a5cedc340d84 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:15.218386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9744591e-347a-4af5-b8be-a5cedc340d84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:15.218929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9744591e-347a-4af5-b8be-a5cedc340d84 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:15.219764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9744591e-347a-4af5-b8be-a5cedc340d84 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:15.221964 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9744591e-347a-4af5-b8be-a5cedc340d84 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:15.233611 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264b74b2-e4a6-49c5-ad41-b9c3d4ba8b17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:15.234316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264b74b2-e4a6-49c5-ad41-b9c3d4ba8b17 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:15.234598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264b74b2-e4a6-49c5-ad41-b9c3d4ba8b17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:15.234877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264b74b2-e4a6-49c5-ad41-b9c3d4ba8b17 admin admin] PATH_INFO: `/v3/projects/cb179de0cdb04915b0477c64ed144107/users/a943238ff42c41ef9bbc877d878af336/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:15.256089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264b74b2-e4a6-49c5-ad41-b9c3d4ba8b17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb179de0cdb04915b0477c64ed144107, user_id=a943238ff42c41ef9bbc877d878af336, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:15.259086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264b74b2-e4a6-49c5-ad41-b9c3d4ba8b17 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:15.353543 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 924/1870] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:49:15 2022] POST /identity/v3/users => generated 289 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:15.357734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9744591e-347a-4af5-b8be-a5cedc340d84 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:15.368523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9744591e-347a-4af5-b8be-a5cedc340d84 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:15.374163 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be18255-b46f-4f76-bc2e-62c281f40f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:15.374943 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be18255-b46f-4f76-bc2e-62c281f40f92 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:15.375288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be18255-b46f-4f76-bc2e-62c281f40f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:15.375572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be18255-b46f-4f76-bc2e-62c281f40f92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:15.376233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be18255-b46f-4f76-bc2e-62c281f40f92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:15.377877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be18255-b46f-4f76-bc2e-62c281f40f92 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:15.378590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 947/1871] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:15 2022] PUT /identity/v3/projects/cb179de0cdb04915b0477c64ed144107/users/a943238ff42c41ef9bbc877d878af336/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:15.384793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b20c60-51a5-4af1-bd34-ddbeb480236e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:15.385595 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b20c60-51a5-4af1-bd34-ddbeb480236e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:15.385968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b20c60-51a5-4af1-bd34-ddbeb480236e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:15.398081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 925/1872] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:15 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:49:15.402025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be18255-b46f-4f76-bc2e-62c281f40f92 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:15.413112 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be18255-b46f-4f76-bc2e-62c281f40f92 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:15.431803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-a2b20c60-51a5-4af1-bd34-ddbeb480236e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:15.433067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a2b20c60-51a5-4af1-bd34-ddbeb480236e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:15.499127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e7c462-c70a-4bf0-a3ad-f23c91c54754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:15.499993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e7c462-c70a-4bf0-a3ad-f23c91c54754 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:15.500278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e7c462-c70a-4bf0-a3ad-f23c91c54754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:15.500551 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e7c462-c70a-4bf0-a3ad-f23c91c54754 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:15.501228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e7c462-c70a-4bf0-a3ad-f23c91c54754 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:15.502966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e7c462-c70a-4bf0-a3ad-f23c91c54754 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:15.508452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 948/1873] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:15.516906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 926/1874] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:15 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:49:15.518024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b20c60-51a5-4af1-bd34-ddbeb480236e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:15.518461 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e7c462-c70a-4bf0-a3ad-f23c91c54754 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:15.520825 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-74e7c462-c70a-4bf0-a3ad-f23c91c54754 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:15.528786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b20c60-51a5-4af1-bd34-ddbeb480236e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:15.534262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787b0a42-03a7-426c-9650-1d93ce8763d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:15.534973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787b0a42-03a7-426c-9650-1d93ce8763d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:15.535248 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787b0a42-03a7-426c-9650-1d93ce8763d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:15.535521 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787b0a42-03a7-426c-9650-1d93ce8763d3 admin admin] PATH_INFO: `/v3/projects/5905682d27bf40ae816e75626ce522f9/users/3c2790a3529843a9b2fd085c8dfc3fff/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:15.550925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787b0a42-03a7-426c-9650-1d93ce8763d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5905682d27bf40ae816e75626ce522f9, user_id=3c2790a3529843a9b2fd085c8dfc3fff, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:15.557358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787b0a42-03a7-426c-9650-1d93ce8763d3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:15.580655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e7c462-c70a-4bf0-a3ad-f23c91c54754 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:15.585095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5365e8a-0b29-4059-b176-d726a1c5b044 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:15.585706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5365e8a-0b29-4059-b176-d726a1c5b044 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:15.586001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5365e8a-0b29-4059-b176-d726a1c5b044 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:15.586293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5365e8a-0b29-4059-b176-d726a1c5b044 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:15.597936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5365e8a-0b29-4059-b176-d726a1c5b044 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:15.599844 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5365e8a-0b29-4059-b176-d726a1c5b044 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:15.613472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 927/1875] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:49:15.644955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:15.650239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 949/1876] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:15 2022] PUT /identity/v3/projects/5905682d27bf40ae816e75626ce522f9/users/3c2790a3529843a9b2fd085c8dfc3fff/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:15.667107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9085d123-8c1c-4414-ae00-831c3050edf4 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:15.667529 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9085d123-8c1c-4414-ae00-831c3050edf4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:15.668038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9085d123-8c1c-4414-ae00-831c3050edf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:15.707034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-9085d123-8c1c-4414-ae00-831c3050edf4 None None] MFA Rules not processed for user `3c2790a3529843a9b2fd085c8dfc3fff`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:15.708576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-9085d123-8c1c-4414-ae00-831c3050edf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:15.803500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 928/1877] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:15 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:15.815452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787b0a42-03a7-426c-9650-1d93ce8763d3 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:15.818462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-787b0a42-03a7-426c-9650-1d93ce8763d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:15.822210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:15.916614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787b0a42-03a7-426c-9650-1d93ce8763d3 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:15.922668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a8cd32-c194-4c28-b896-522538f17d4a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f3b3c7ebc08496e970f8f25fd6395e4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:15.923370 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a8cd32-c194-4c28-b896-522538f17d4a service glance] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:15.923710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a8cd32-c194-4c28-b896-522538f17d4a service glance] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:15.924269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a8cd32-c194-4c28-b896-522538f17d4a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:15.937648 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a8cd32-c194-4c28-b896-522538f17d4a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:15.939270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a8cd32-c194-4c28-b896-522538f17d4a service glance] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:15.950468 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 950/1878] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:15 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:49:17.519464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:17.688826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d45674-137b-4e50-8012-c74554bd333e None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:17.689346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d45674-137b-4e50-8012-c74554bd333e None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:17.689426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d45674-137b-4e50-8012-c74554bd333e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:17.694795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:17.730851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-a9d45674-137b-4e50-8012-c74554bd333e None None] MFA Rules not processed for user `a943238ff42c41ef9bbc877d878af336`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:17.732234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a9d45674-137b-4e50-8012-c74554bd333e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:17.856976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 929/1879] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:17 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:17.858744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a8cd32-c194-4c28-b896-522538f17d4a service glance] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:17.968717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a8cd32-c194-4c28-b896-522538f17d4a service glance] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:17.973646 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eb5886-2f98-4c32-82cb-41765a62bad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:17.974496 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eb5886-2f98-4c32-82cb-41765a62bad7 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:17.974780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eb5886-2f98-4c32-82cb-41765a62bad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:17.975055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eb5886-2f98-4c32-82cb-41765a62bad7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:17.975693 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eb5886-2f98-4c32-82cb-41765a62bad7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:17.978196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eb5886-2f98-4c32-82cb-41765a62bad7 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:18.062895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 951/1880] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:49:17 2022] POST /identity/v3/projects => generated 369 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:18.068085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d45674-137b-4e50-8012-c74554bd333e None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:18.081005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d45674-137b-4e50-8012-c74554bd333e None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:18.148658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538d5f1f-79c0-409d-a920-1dcfb3c0ec66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:18.149284 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538d5f1f-79c0-409d-a920-1dcfb3c0ec66 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:18.149559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538d5f1f-79c0-409d-a920-1dcfb3c0ec66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:18.149827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538d5f1f-79c0-409d-a920-1dcfb3c0ec66 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:18.150577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538d5f1f-79c0-409d-a920-1dcfb3c0ec66 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:18.152486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538d5f1f-79c0-409d-a920-1dcfb3c0ec66 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:18.251975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 930/1881] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:49:18 2022] POST /identity/v3/users => generated 291 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:49:18.256897 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eb5886-2f98-4c32-82cb-41765a62bad7 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:18.272504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eb5886-2f98-4c32-82cb-41765a62bad7 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:18.278049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc235fc-0992-4117-a493-b947342f4c8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:18.279202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc235fc-0992-4117-a493-b947342f4c8f admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:18.279202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc235fc-0992-4117-a493-b947342f4c8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:18.279440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc235fc-0992-4117-a493-b947342f4c8f admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:18.280134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc235fc-0992-4117-a493-b947342f4c8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:18.282705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc235fc-0992-4117-a493-b947342f4c8f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:18.309211 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 952/1882] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:18 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:49:18.322604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538d5f1f-79c0-409d-a920-1dcfb3c0ec66 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:18.335508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538d5f1f-79c0-409d-a920-1dcfb3c0ec66 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:18.343818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09e2ef2-ae2e-4462-b1b6-d69c210043e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:18.354134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e2ef2-ae2e-4462-b1b6-d69c210043e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:18.354134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e2ef2-ae2e-4462-b1b6-d69c210043e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:18.354134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09e2ef2-ae2e-4462-b1b6-d69c210043e7 admin admin] PATH_INFO: `/v3/projects/33f8bf4f833b45d98b9c90257290da2d/users/ecf60e9e3a754cd0b32131c7ab045965/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:18.397402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09e2ef2-ae2e-4462-b1b6-d69c210043e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33f8bf4f833b45d98b9c90257290da2d, user_id=ecf60e9e3a754cd0b32131c7ab045965, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:18.403911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09e2ef2-ae2e-4462-b1b6-d69c210043e7 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:18.551043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 931/1883] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:18 2022] PUT /identity/v3/projects/33f8bf4f833b45d98b9c90257290da2d/users/ecf60e9e3a754cd0b32131c7ab045965/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:49:18.552968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc235fc-0992-4117-a493-b947342f4c8f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:18.570669 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc235fc-0992-4117-a493-b947342f4c8f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:18.662624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65821a77-b82b-4334-bb0b-1f96a5a29af8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:18.663798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65821a77-b82b-4334-bb0b-1f96a5a29af8 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:18.664398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65821a77-b82b-4334-bb0b-1f96a5a29af8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:18.664820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65821a77-b82b-4334-bb0b-1f96a5a29af8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:18.665927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65821a77-b82b-4334-bb0b-1f96a5a29af8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:18.667358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65821a77-b82b-4334-bb0b-1f96a5a29af8 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:18.682640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 953/1884] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:18 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:49:18.699751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09e2ef2-ae2e-4462-b1b6-d69c210043e7 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:18.714706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09e2ef2-ae2e-4462-b1b6-d69c210043e7 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:18.720244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1168b77-529d-4f5d-ac37-23531a93a595 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:18.720994 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1168b77-529d-4f5d-ac37-23531a93a595 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:18.721341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1168b77-529d-4f5d-ac37-23531a93a595 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:18.721675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1168b77-529d-4f5d-ac37-23531a93a595 admin admin] PATH_INFO: `/v3/projects/33f8bf4f833b45d98b9c90257290da2d/users/ecf60e9e3a754cd0b32131c7ab045965/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:18.724706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1168b77-529d-4f5d-ac37-23531a93a595 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33f8bf4f833b45d98b9c90257290da2d, user_id=ecf60e9e3a754cd0b32131c7ab045965, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:18.728127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1168b77-529d-4f5d-ac37-23531a93a595 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:18.742325 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 932/1885] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:18 2022] PUT /identity/v3/projects/33f8bf4f833b45d98b9c90257290da2d/users/ecf60e9e3a754cd0b32131c7ab045965/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:19.004801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:20.279150 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:20.558891 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:20.759818 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b961da95-ac99-4468-9f01-bd3ef911741b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:20.760107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b961da95-ac99-4468-9f01-bd3ef911741b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:20.760384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b961da95-ac99-4468-9f01-bd3ef911741b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:20.797970 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-b961da95-ac99-4468-9f01-bd3ef911741b None None] MFA Rules not processed for user `ecf60e9e3a754cd0b32131c7ab045965`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:20.799396 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b961da95-ac99-4468-9f01-bd3ef911741b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:20.894532 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 954/1886] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:20 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:20.906101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1168b77-529d-4f5d-ac37-23531a93a595 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:20.909805 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a1168b77-529d-4f5d-ac37-23531a93a595 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:20.978702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1168b77-529d-4f5d-ac37-23531a93a595 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:20.982626 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4e66e8-b044-46e9-8d81-f50f9031c591 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:20.983274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4e66e8-b044-46e9-8d81-f50f9031c591 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:20.983550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4e66e8-b044-46e9-8d81-f50f9031c591 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:20.983846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4e66e8-b044-46e9-8d81-f50f9031c591 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:21.055403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4e66e8-b044-46e9-8d81-f50f9031c591 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:21.056847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4e66e8-b044-46e9-8d81-f50f9031c591 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:21.080132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 933/1887] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:20 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:49:21.931436 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=83,keystone:INSERT=2,keystone:DELETE=15 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:22.180181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:23.290075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:23.410203 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:24.363450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:24.513470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:25.072698 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:25.123279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b961da95-ac99-4468-9f01-bd3ef911741b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:25.124768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b961da95-ac99-4468-9f01-bd3ef911741b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:25.137625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b961da95-ac99-4468-9f01-bd3ef911741b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:25.141427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecab45f2-a310-46ad-b283-8f8210b379b7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:25.142032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecab45f2-a310-46ad-b283-8f8210b379b7 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:25.142347 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecab45f2-a310-46ad-b283-8f8210b379b7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:25.142621 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecab45f2-a310-46ad-b283-8f8210b379b7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:25.152365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecab45f2-a310-46ad-b283-8f8210b379b7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:25.153807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecab45f2-a310-46ad-b283-8f8210b379b7 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:25.164328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 955/1888] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:25 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:49:25.225854 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:25.633552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:25.815959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:26.365072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:26.907322 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:27.341441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:37.616012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=15 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:53.463280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f02e921-b54e-40df-95cc-ae75295b96ea None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:53.464386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f02e921-b54e-40df-95cc-ae75295b96ea None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:53.464386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f02e921-b54e-40df-95cc-ae75295b96ea None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:53.465994 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 934/1889] 10.222.0.65 () {54 vars in 921 bytes} [Wed Aug 24 03:49:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:49:53.478836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3ee51d-6509-4aae-8688-abdb5128924d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:53.479504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3ee51d-6509-4aae-8688-abdb5128924d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:53.479504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3ee51d-6509-4aae-8688-abdb5128924d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:53.500115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-df3ee51d-6509-4aae-8688-abdb5128924d None None] MFA Rules not processed for user `4b347c9c89a64168b51c439b4a92d647`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:53.501446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-df3ee51d-6509-4aae-8688-abdb5128924d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:53.600182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 956/1890] 10.222.0.65 () {58 vars in 1018 bytes} [Wed Aug 24 03:49:53 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:53.611811 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f02e921-b54e-40df-95cc-ae75295b96ea None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:53.619749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-8f02e921-b54e-40df-95cc-ae75295b96ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:53.636010 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f02e921-b54e-40df-95cc-ae75295b96ea None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:53.642505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9398766-339d-4f70-9f94-0ef6a273b308 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:53.648055 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9398766-339d-4f70-9f94-0ef6a273b308 service placement] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:53.648055 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9398766-339d-4f70-9f94-0ef6a273b308 service placement] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:53.648055 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9398766-339d-4f70-9f94-0ef6a273b308 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:53.653035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9398766-339d-4f70-9f94-0ef6a273b308 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:53.654666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9398766-339d-4f70-9f94-0ef6a273b308 service placement] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:53.666424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 935/1891] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:49:54.373295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3ee51d-6509-4aae-8688-abdb5128924d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:54.446087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3ee51d-6509-4aae-8688-abdb5128924d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:54.450882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:54.452458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:54.452756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:54.453037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] PATH_INFO: `/v3/users/a943238ff42c41ef9bbc877d878af336` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:54.454272 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a943238ff42c41ef9bbc877d878af336)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:54.456245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:54.657455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a943238ff42c41ef9bbc877d878af336'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:54.658624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a943238ff42c41ef9bbc877d878af336'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:54.665427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a943238ff42c41ef9bbc877d878af336'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:54.666137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a943238ff42c41ef9bbc877d878af336'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:54.733108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a943238ff42c41ef9bbc877d878af336'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:54.752460 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 957/1892] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:49:54 2022] DELETE /identity/v3/users/a943238ff42c41ef9bbc877d878af336 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:55.035806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9398766-339d-4f70-9f94-0ef6a273b308 service placement] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:55.051501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9398766-339d-4f70-9f94-0ef6a273b308 service placement] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:55.137092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:55.137855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:55.138320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:55.138915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] PATH_INFO: `/v3/projects/cb179de0cdb04915b0477c64ed144107` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:55.140188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb179de0cdb04915b0477c64ed144107)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:55.142248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:55.311766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb179de0cdb04915b0477c64ed144107', 'request_id': 'req-7f13d2b4-519d-4ca7-b53f-532528daca25'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:55.317567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb179de0cdb04915b0477c64ed144107', 'request_id': 'req-7f13d2b4-519d-4ca7-b53f-532528daca25'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:55.323870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb179de0cdb04915b0477c64ed144107', 'request_id': 'req-7f13d2b4-519d-4ca7-b53f-532528daca25'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:55.403440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] The token cache is being invalidate because project cb179de0cdb04915b0477c64ed144107 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:49:55.403788 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:55.404415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:55.406245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 936/1893] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:49:55 2022] DELETE /identity/v3/projects/cb179de0cdb04915b0477c64ed144107 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:55.417236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:55.417236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:55.433414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767346a0-64fb-4efe-a114-30bcef0d1c65 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:55.447807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:55.501756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb7a25d-aada-4a8a-a10f-6f6ae0d2a650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:55.502457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb7a25d-aada-4a8a-a10f-6f6ae0d2a650 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:55.502737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb7a25d-aada-4a8a-a10f-6f6ae0d2a650 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:55.503018 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb7a25d-aada-4a8a-a10f-6f6ae0d2a650 admin admin] PATH_INFO: `/v3/users/ecf60e9e3a754cd0b32131c7ab045965` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:55.504263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb7a25d-aada-4a8a-a10f-6f6ae0d2a650 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ecf60e9e3a754cd0b32131c7ab045965)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:55.506437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb7a25d-aada-4a8a-a10f-6f6ae0d2a650 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:55.756509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2bb7a25d-aada-4a8a-a10f-6f6ae0d2a650 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ecf60e9e3a754cd0b32131c7ab045965'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:55.757504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2bb7a25d-aada-4a8a-a10f-6f6ae0d2a650 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ecf60e9e3a754cd0b32131c7ab045965'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:55.767371 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2bb7a25d-aada-4a8a-a10f-6f6ae0d2a650 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ecf60e9e3a754cd0b32131c7ab045965'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:55.768077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2bb7a25d-aada-4a8a-a10f-6f6ae0d2a650 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ecf60e9e3a754cd0b32131c7ab045965'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:55.849097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2bb7a25d-aada-4a8a-a10f-6f6ae0d2a650 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ecf60e9e3a754cd0b32131c7ab045965'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:55.870416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 958/1894] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:49:55 2022] DELETE /identity/v3/users/ecf60e9e3a754cd0b32131c7ab045965 => generated 0 bytes in 456 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:56.363596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:56.365481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:56.378474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f13d2b4-519d-4ca7-b53f-532528daca25 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:56.447748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905c8388-b788-4eca-95f3-9d249a34e32b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:56.448401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905c8388-b788-4eca-95f3-9d249a34e32b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:56.448677 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905c8388-b788-4eca-95f3-9d249a34e32b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:56.448946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905c8388-b788-4eca-95f3-9d249a34e32b admin admin] PATH_INFO: `/v3/projects/33f8bf4f833b45d98b9c90257290da2d` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:56.450130 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905c8388-b788-4eca-95f3-9d249a34e32b admin admin] RBAC: Authorizing `identity:delete_project(project_id=33f8bf4f833b45d98b9c90257290da2d)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:56.452095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905c8388-b788-4eca-95f3-9d249a34e32b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:56.479919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:56.599900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-905c8388-b788-4eca-95f3-9d249a34e32b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33f8bf4f833b45d98b9c90257290da2d', 'request_id': 'req-905c8388-b788-4eca-95f3-9d249a34e32b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:56.606495 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-905c8388-b788-4eca-95f3-9d249a34e32b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33f8bf4f833b45d98b9c90257290da2d', 'request_id': 'req-905c8388-b788-4eca-95f3-9d249a34e32b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:56.613407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-905c8388-b788-4eca-95f3-9d249a34e32b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33f8bf4f833b45d98b9c90257290da2d', 'request_id': 'req-905c8388-b788-4eca-95f3-9d249a34e32b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:56.757765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-905c8388-b788-4eca-95f3-9d249a34e32b admin admin] The token cache is being invalidate because project 33f8bf4f833b45d98b9c90257290da2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:49:56.758062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-905c8388-b788-4eca-95f3-9d249a34e32b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:56.763096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-905c8388-b788-4eca-95f3-9d249a34e32b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:56.763096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 937/1895] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:49:56 2022] DELETE /identity/v3/projects/33f8bf4f833b45d98b9c90257290da2d => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:56.766935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cea028-8dff-42cd-9997-fa399851c942 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:56.767494 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cea028-8dff-42cd-9997-fa399851c942 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:56.767873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cea028-8dff-42cd-9997-fa399851c942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:56.789644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-53cea028-8dff-42cd-9997-fa399851c942 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:56.790981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-53cea028-8dff-42cd-9997-fa399851c942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:56.857304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 959/1896] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:56.866355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4575ba3-e962-47ce-9bcd-7305b9a66a40 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:56.866651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4575ba3-e962-47ce-9bcd-7305b9a66a40 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:56.866925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4575ba3-e962-47ce-9bcd-7305b9a66a40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:56.889260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-b4575ba3-e962-47ce-9bcd-7305b9a66a40 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:56.890578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b4575ba3-e962-47ce-9bcd-7305b9a66a40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:56.899503 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 938/1897] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:56.904057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cea028-8dff-42cd-9997-fa399851c942 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:56.915779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cea028-8dff-42cd-9997-fa399851c942 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:56.924111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48c2630-d66d-4dee-91bc-2c2f15a6b08a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:56.924741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48c2630-d66d-4dee-91bc-2c2f15a6b08a admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:56.925543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48c2630-d66d-4dee-91bc-2c2f15a6b08a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:56.925543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48c2630-d66d-4dee-91bc-2c2f15a6b08a admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:56.927030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48c2630-d66d-4dee-91bc-2c2f15a6b08a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:49:56.927030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48c2630-d66d-4dee-91bc-2c2f15a6b08a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:56.929105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48c2630-d66d-4dee-91bc-2c2f15a6b08a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:56.942875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 960/1898] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:49:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:49:56.947541 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4575ba3-e962-47ce-9bcd-7305b9a66a40 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:56.957052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4575ba3-e962-47ce-9bcd-7305b9a66a40 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:56.962020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967e9cc4-a7ef-42ea-b358-8ac8f08ce441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:56.962676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967e9cc4-a7ef-42ea-b358-8ac8f08ce441 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:56.962978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967e9cc4-a7ef-42ea-b358-8ac8f08ce441 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:56.963221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967e9cc4-a7ef-42ea-b358-8ac8f08ce441 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:56.963859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967e9cc4-a7ef-42ea-b358-8ac8f08ce441 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:56.965819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967e9cc4-a7ef-42ea-b358-8ac8f08ce441 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:57.074087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 939/1899] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:49:56 2022] POST /identity/v3/projects => generated 383 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:57.078324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48c2630-d66d-4dee-91bc-2c2f15a6b08a admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:57.093113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48c2630-d66d-4dee-91bc-2c2f15a6b08a admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:57.116565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:57.160834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec91fc8-1b77-4f89-a3b5-b8ebd3cb1e57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:57.161493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec91fc8-1b77-4f89-a3b5-b8ebd3cb1e57 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:57.161772 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec91fc8-1b77-4f89-a3b5-b8ebd3cb1e57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:57.162047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec91fc8-1b77-4f89-a3b5-b8ebd3cb1e57 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:57.162731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec91fc8-1b77-4f89-a3b5-b8ebd3cb1e57 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:57.164671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec91fc8-1b77-4f89-a3b5-b8ebd3cb1e57 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:57.275384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 961/1900] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:49:57 2022] POST /identity/v3/users => generated 292 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:57.276673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967e9cc4-a7ef-42ea-b358-8ac8f08ce441 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:57.288459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967e9cc4-a7ef-42ea-b358-8ac8f08ce441 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:57.293424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0dda8c-cbd1-4efc-abb4-43cc49a976f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:57.294001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0dda8c-cbd1-4efc-abb4-43cc49a976f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:57.294329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0dda8c-cbd1-4efc-abb4-43cc49a976f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:57.294601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0dda8c-cbd1-4efc-abb4-43cc49a976f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:57.295255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0dda8c-cbd1-4efc-abb4-43cc49a976f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:57.296879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0dda8c-cbd1-4efc-abb4-43cc49a976f8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:57.312849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 940/1901] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:57 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:49:57.314540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec91fc8-1b77-4f89-a3b5-b8ebd3cb1e57 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:57.329049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec91fc8-1b77-4f89-a3b5-b8ebd3cb1e57 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:57.351869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2669159d-9b2e-4419-8be1-d942d8ece232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:57.351869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2669159d-9b2e-4419-8be1-d942d8ece232 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:57.351869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2669159d-9b2e-4419-8be1-d942d8ece232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:57.351869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2669159d-9b2e-4419-8be1-d942d8ece232 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:57.352691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2669159d-9b2e-4419-8be1-d942d8ece232 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:57.354106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2669159d-9b2e-4419-8be1-d942d8ece232 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:57.385927 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0dda8c-cbd1-4efc-abb4-43cc49a976f8 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:57.389548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 962/1902] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:57 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:49:57.400867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0dda8c-cbd1-4efc-abb4-43cc49a976f8 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:57.406769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78ee8c4-a2fd-49bd-b30a-fba7549ba96a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:57.407441 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78ee8c4-a2fd-49bd-b30a-fba7549ba96a admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:57.408492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78ee8c4-a2fd-49bd-b30a-fba7549ba96a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:57.409125 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78ee8c4-a2fd-49bd-b30a-fba7549ba96a admin admin] PATH_INFO: `/v3/projects/7585c97131fa4697abfa45e1d002d1b9/users/17c69ff6f9314c3387461af42eba52f8/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:57.432107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78ee8c4-a2fd-49bd-b30a-fba7549ba96a admin admin] RBAC: Authorizing `identity:create_grant(project_id=7585c97131fa4697abfa45e1d002d1b9, user_id=17c69ff6f9314c3387461af42eba52f8, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:57.438208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78ee8c4-a2fd-49bd-b30a-fba7549ba96a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:57.525325 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 941/1903] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:57 2022] PUT /identity/v3/projects/7585c97131fa4697abfa45e1d002d1b9/users/17c69ff6f9314c3387461af42eba52f8/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:57.537111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2806cb-4929-404e-9935-c19d57022f2a None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:57.537405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2806cb-4929-404e-9935-c19d57022f2a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:57.537680 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2806cb-4929-404e-9935-c19d57022f2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:57.571622 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-9f2806cb-4929-404e-9935-c19d57022f2a None None] MFA Rules not processed for user `17c69ff6f9314c3387461af42eba52f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:57.573335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9f2806cb-4929-404e-9935-c19d57022f2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:57.655659 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 963/1904] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:57.660188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78ee8c4-a2fd-49bd-b30a-fba7549ba96a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:57.735089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78ee8c4-a2fd-49bd-b30a-fba7549ba96a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:57.740783 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8882f0b-eb02-4390-a67c-dd2dcacb9849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:57.741395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8882f0b-eb02-4390-a67c-dd2dcacb9849 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:57.741668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8882f0b-eb02-4390-a67c-dd2dcacb9849 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:57.741950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8882f0b-eb02-4390-a67c-dd2dcacb9849 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:57.742634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8882f0b-eb02-4390-a67c-dd2dcacb9849 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:57.744618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8882f0b-eb02-4390-a67c-dd2dcacb9849 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:57.858707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 942/1905] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:49:57 2022] POST /identity/v3/projects => generated 385 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:57.860152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2806cb-4929-404e-9935-c19d57022f2a None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:57.872864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2806cb-4929-404e-9935-c19d57022f2a None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:57.937170 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1d26eb-e675-45c7-b56d-9753f641d358 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:57.937707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1d26eb-e675-45c7-b56d-9753f641d358 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:57.937992 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1d26eb-e675-45c7-b56d-9753f641d358 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:57.938305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1d26eb-e675-45c7-b56d-9753f641d358 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:57.938958 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1d26eb-e675-45c7-b56d-9753f641d358 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:57.940898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1d26eb-e675-45c7-b56d-9753f641d358 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:58.080826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 964/1906] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:49:57 2022] POST /identity/v3/users => generated 299 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:49:58.085297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8882f0b-eb02-4390-a67c-dd2dcacb9849 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:58.099030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8882f0b-eb02-4390-a67c-dd2dcacb9849 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:58.105325 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126916fd-3382-45e7-b937-60e2bd79b22b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:58.106222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126916fd-3382-45e7-b937-60e2bd79b22b admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:58.106858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126916fd-3382-45e7-b937-60e2bd79b22b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:58.106858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126916fd-3382-45e7-b937-60e2bd79b22b admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:58.107846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126916fd-3382-45e7-b937-60e2bd79b22b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:58.109884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126916fd-3382-45e7-b937-60e2bd79b22b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:58.133306 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 943/1907] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:58 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:49:58.137603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1d26eb-e675-45c7-b56d-9753f641d358 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:58.147848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1d26eb-e675-45c7-b56d-9753f641d358 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:58.152744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719ec8af-e146-467d-bcee-2cdf688fef5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:58.153419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719ec8af-e146-467d-bcee-2cdf688fef5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:58.153696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719ec8af-e146-467d-bcee-2cdf688fef5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:58.153977 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719ec8af-e146-467d-bcee-2cdf688fef5f admin admin] PATH_INFO: `/v3/projects/1020dc139dd0423bbb5c70fd91e83da4/users/b189506adf4c41409cf878fe7df0bb91/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:58.170728 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719ec8af-e146-467d-bcee-2cdf688fef5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=1020dc139dd0423bbb5c70fd91e83da4, user_id=b189506adf4c41409cf878fe7df0bb91, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:58.173923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719ec8af-e146-467d-bcee-2cdf688fef5f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:58.177926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:58.293348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 965/1908] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:58 2022] PUT /identity/v3/projects/1020dc139dd0423bbb5c70fd91e83da4/users/b189506adf4c41409cf878fe7df0bb91/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:58.297470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126916fd-3382-45e7-b937-60e2bd79b22b admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:58.309999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126916fd-3382-45e7-b937-60e2bd79b22b admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:58.326543 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:58.372293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091a91e1-8c57-4eb2-bf04-b272cde6ae1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:58.372900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091a91e1-8c57-4eb2-bf04-b272cde6ae1c admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:58.373173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091a91e1-8c57-4eb2-bf04-b272cde6ae1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:58.373438 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091a91e1-8c57-4eb2-bf04-b272cde6ae1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:58.374077 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091a91e1-8c57-4eb2-bf04-b272cde6ae1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:58.375700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091a91e1-8c57-4eb2-bf04-b272cde6ae1c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:58.387242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 944/1909] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:49:58 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:49:58.391828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719ec8af-e146-467d-bcee-2cdf688fef5f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:58.412932 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719ec8af-e146-467d-bcee-2cdf688fef5f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:58.418551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb07bd64-4dd3-4aa9-b5c8-aca2f80f42c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:58.419532 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb07bd64-4dd3-4aa9-b5c8-aca2f80f42c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:58.422203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb07bd64-4dd3-4aa9-b5c8-aca2f80f42c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:58.422546 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb07bd64-4dd3-4aa9-b5c8-aca2f80f42c3 admin admin] PATH_INFO: `/v3/projects/1020dc139dd0423bbb5c70fd91e83da4/users/b189506adf4c41409cf878fe7df0bb91/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:58.424896 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb07bd64-4dd3-4aa9-b5c8-aca2f80f42c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1020dc139dd0423bbb5c70fd91e83da4, user_id=b189506adf4c41409cf878fe7df0bb91, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:58.428884 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb07bd64-4dd3-4aa9-b5c8-aca2f80f42c3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:58.430397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091a91e1-8c57-4eb2-bf04-b272cde6ae1c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:58.433006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-091a91e1-8c57-4eb2-bf04-b272cde6ae1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:58.443964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 966/1910] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:49:58 2022] PUT /identity/v3/projects/1020dc139dd0423bbb5c70fd91e83da4/users/b189506adf4c41409cf878fe7df0bb91/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:58.448660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091a91e1-8c57-4eb2-bf04-b272cde6ae1c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:58.455910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bc1644-4725-4e27-b4bd-7ef6b679e4c5 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:58.456692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bc1644-4725-4e27-b4bd-7ef6b679e4c5 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:58.457994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bc1644-4725-4e27-b4bd-7ef6b679e4c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:58.501017 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-02bc1644-4725-4e27-b4bd-7ef6b679e4c5 None None] MFA Rules not processed for user `b189506adf4c41409cf878fe7df0bb91`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:58.502295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-02bc1644-4725-4e27-b4bd-7ef6b679e4c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:58.531879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:58.534004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:58.534004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:58.534004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] PATH_INFO: `/v3/users/1d05c440cf484828b3589a671f69b503` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:58.534343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d05c440cf484828b3589a671f69b503)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:58.536994 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:58.580294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 967/1911] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:58 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:49:58.592660 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bc1644-4725-4e27-b4bd-7ef6b679e4c5 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:58.594087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-02bc1644-4725-4e27-b4bd-7ef6b679e4c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:58.654017 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bc1644-4725-4e27-b4bd-7ef6b679e4c5 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:58.657713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541fd655-84b8-482c-8e93-e5f6eabb9cf6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:58.658397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541fd655-84b8-482c-8e93-e5f6eabb9cf6 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:58.658951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541fd655-84b8-482c-8e93-e5f6eabb9cf6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:58.658951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541fd655-84b8-482c-8e93-e5f6eabb9cf6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:58.721758 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541fd655-84b8-482c-8e93-e5f6eabb9cf6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:58.723761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541fd655-84b8-482c-8e93-e5f6eabb9cf6 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:58.733875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 968/1912] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:49:58.824242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d05c440cf484828b3589a671f69b503'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:58.824845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d05c440cf484828b3589a671f69b503'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:58.832512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d05c440cf484828b3589a671f69b503'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:58.845510 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d05c440cf484828b3589a671f69b503'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:58.934906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d05c440cf484828b3589a671f69b503'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:58.942104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 945/1913] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:49:58 2022] DELETE /identity/v3/users/1d05c440cf484828b3589a671f69b503 => generated 0 bytes in 512 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:49:58.946804 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41171e0f-9923-4888-84a0-2d8d072a9bfa None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:58.947349 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41171e0f-9923-4888-84a0-2d8d072a9bfa None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:58.947349 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41171e0f-9923-4888-84a0-2d8d072a9bfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:58.967377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-41171e0f-9923-4888-84a0-2d8d072a9bfa None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:49:58.968605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-41171e0f-9923-4888-84a0-2d8d072a9bfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:58.985807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:59.032208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 969/1914] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:49:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:49:59.044109 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:59.046136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:59.089082 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:49:59.111321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7628cd-ff67-4ffc-926d-4ec1d6f75809 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:59.116036 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e74b75-7a06-4b48-91dd-fad9e42e0dfb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:59.116643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e74b75-7a06-4b48-91dd-fad9e42e0dfb service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:59.117082 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e74b75-7a06-4b48-91dd-fad9e42e0dfb service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:59.117182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e74b75-7a06-4b48-91dd-fad9e42e0dfb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:59.126682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e74b75-7a06-4b48-91dd-fad9e42e0dfb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:59.128109 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e74b75-7a06-4b48-91dd-fad9e42e0dfb service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:59.138452 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 946/1915] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:49:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:49:59.638059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41171e0f-9923-4888-84a0-2d8d072a9bfa None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:59.640264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-41171e0f-9923-4888-84a0-2d8d072a9bfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:49:59.656040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41171e0f-9923-4888-84a0-2d8d072a9bfa None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:49:59.662593 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:49:59.663224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:49:59.663502 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:49:59.663774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] PATH_INFO: `/v3/projects/5d3925a8b5fe44868a8db62237ea42ef` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:49:59.667196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d3925a8b5fe44868a8db62237ea42ef)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:49:59.669273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:49:59.796133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d3925a8b5fe44868a8db62237ea42ef', 'request_id': 'req-6709cd2e-807c-4802-991f-f79d90f91caf'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:59.807008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d3925a8b5fe44868a8db62237ea42ef', 'request_id': 'req-6709cd2e-807c-4802-991f-f79d90f91caf'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:59.907933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d3925a8b5fe44868a8db62237ea42ef', 'request_id': 'req-6709cd2e-807c-4802-991f-f79d90f91caf'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:59.922709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] The token cache is being invalidate because project 5d3925a8b5fe44868a8db62237ea42ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:49:59.923007 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:59.924028 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:49:59.926439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 970/1916] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:49:59 2022] DELETE /identity/v3/projects/5d3925a8b5fe44868a8db62237ea42ef => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:49:59.930632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e74b75-7a06-4b48-91dd-fad9e42e0dfb service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:49:59.941165 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e74b75-7a06-4b48-91dd-fad9e42e0dfb service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:00.005486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15abe2e6-d09f-4fa7-94b1-d63a94f1d86e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:00.006137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15abe2e6-d09f-4fa7-94b1-d63a94f1d86e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:00.006446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15abe2e6-d09f-4fa7-94b1-d63a94f1d86e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:00.006759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15abe2e6-d09f-4fa7-94b1-d63a94f1d86e admin admin] PATH_INFO: `/v3/users/de2ae72bc3b44c489efd46390f796bce` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:00.007922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15abe2e6-d09f-4fa7-94b1-d63a94f1d86e admin admin] RBAC: Authorizing `identity:delete_user(user_id=de2ae72bc3b44c489efd46390f796bce)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:00.009853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15abe2e6-d09f-4fa7-94b1-d63a94f1d86e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:00.195821 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-15abe2e6-d09f-4fa7-94b1-d63a94f1d86e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de2ae72bc3b44c489efd46390f796bce'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:00.196488 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-15abe2e6-d09f-4fa7-94b1-d63a94f1d86e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de2ae72bc3b44c489efd46390f796bce'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:00.203974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-15abe2e6-d09f-4fa7-94b1-d63a94f1d86e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de2ae72bc3b44c489efd46390f796bce'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:00.216961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-15abe2e6-d09f-4fa7-94b1-d63a94f1d86e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de2ae72bc3b44c489efd46390f796bce'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:00.288567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-15abe2e6-d09f-4fa7-94b1-d63a94f1d86e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de2ae72bc3b44c489efd46390f796bce'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:00.295244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 947/1917] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:49:59 2022] DELETE /identity/v3/users/de2ae72bc3b44c489efd46390f796bce => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:00.728690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:00.728690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:00.744915 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6709cd2e-807c-4802-991f-f79d90f91caf admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:00.818084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:00.818751 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:00.824571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:00.824571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee admin admin] PATH_INFO: `/v3/projects/d8ee3b88c6564ba6901d88e720d32203` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:00.824571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=d8ee3b88c6564ba6901d88e720d32203)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:00.824571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:00.978663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:00.981376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd8ee3b88c6564ba6901d88e720d32203', 'request_id': 'req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:00.990436 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd8ee3b88c6564ba6901d88e720d32203', 'request_id': 'req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:01.077539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd8ee3b88c6564ba6901d88e720d32203', 'request_id': 'req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:01.092548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee admin admin] The token cache is being invalidate because project d8ee3b88c6564ba6901d88e720d32203 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:01.092871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:01.093509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9187f20f-ea48-4ae3-87e1-b9b1542f82ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:01.095282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 971/1918] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:00 2022] DELETE /identity/v3/projects/d8ee3b88c6564ba6901d88e720d32203 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:01.100342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca599f-93c9-4bb6-8da3-c4f7d0fde494 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:01.100902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca599f-93c9-4bb6-8da3-c4f7d0fde494 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:01.101178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca599f-93c9-4bb6-8da3-c4f7d0fde494 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:01.125858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-62ca599f-93c9-4bb6-8da3-c4f7d0fde494 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:01.127617 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-62ca599f-93c9-4bb6-8da3-c4f7d0fde494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:01.188647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:01.201758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 948/1919] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:01.218471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7433e174-0fb2-462c-b517-2b1de4b4f9ea None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:01.218671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7433e174-0fb2-462c-b517-2b1de4b4f9ea None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:01.218931 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7433e174-0fb2-462c-b517-2b1de4b4f9ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:01.240334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-7433e174-0fb2-462c-b517-2b1de4b4f9ea None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:01.241573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7433e174-0fb2-462c-b517-2b1de4b4f9ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:01.249535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 972/1920] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:01.254583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca599f-93c9-4bb6-8da3-c4f7d0fde494 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:01.265292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca599f-93c9-4bb6-8da3-c4f7d0fde494 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:01.270817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b6e077-865e-4584-820a-f50de10472a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:01.272230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b6e077-865e-4584-820a-f50de10472a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:01.272230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b6e077-865e-4584-820a-f50de10472a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:01.272230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b6e077-865e-4584-820a-f50de10472a0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:01.272573 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b6e077-865e-4584-820a-f50de10472a0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:50:01.272865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b6e077-865e-4584-820a-f50de10472a0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:01.274480 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b6e077-865e-4584-820a-f50de10472a0 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:01.286233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 949/1921] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:50:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:50:01.290629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7433e174-0fb2-462c-b517-2b1de4b4f9ea None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:01.299953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7433e174-0fb2-462c-b517-2b1de4b4f9ea None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:01.305589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e9c2a5-d28d-44a7-9c0f-d44c2849e6fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:01.306218 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e9c2a5-d28d-44a7-9c0f-d44c2849e6fb admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:01.306496 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e9c2a5-d28d-44a7-9c0f-d44c2849e6fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:01.306763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e9c2a5-d28d-44a7-9c0f-d44c2849e6fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:01.307402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e9c2a5-d28d-44a7-9c0f-d44c2849e6fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:01.309296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e9c2a5-d28d-44a7-9c0f-d44c2849e6fb admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:01.501633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 973/1922] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:01 2022] POST /identity/v3/projects => generated 385 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:01.502897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b6e077-865e-4584-820a-f50de10472a0 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:01.514140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b6e077-865e-4584-820a-f50de10472a0 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:01.574028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7ebde6-08f1-45e2-a74d-6675ce98b216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:01.574664 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7ebde6-08f1-45e2-a74d-6675ce98b216 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:01.574939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7ebde6-08f1-45e2-a74d-6675ce98b216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:01.575209 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7ebde6-08f1-45e2-a74d-6675ce98b216 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:01.575853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7ebde6-08f1-45e2-a74d-6675ce98b216 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:01.577773 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7ebde6-08f1-45e2-a74d-6675ce98b216 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:01.709314 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 950/1923] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:01 2022] POST /identity/v3/users => generated 293 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:01.712682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e9c2a5-d28d-44a7-9c0f-d44c2849e6fb admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:01.724536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e9c2a5-d28d-44a7-9c0f-d44c2849e6fb admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:01.729274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abfd8b3-d0b6-484b-b9fa-87a3bba59efc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:01.729903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abfd8b3-d0b6-484b-b9fa-87a3bba59efc admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:01.730169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abfd8b3-d0b6-484b-b9fa-87a3bba59efc admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:01.730477 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abfd8b3-d0b6-484b-b9fa-87a3bba59efc admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:01.731138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abfd8b3-d0b6-484b-b9fa-87a3bba59efc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:01.732754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abfd8b3-d0b6-484b-b9fa-87a3bba59efc admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:01.745059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 974/1924] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:01 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:01.748721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7ebde6-08f1-45e2-a74d-6675ce98b216 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:01.759921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7ebde6-08f1-45e2-a74d-6675ce98b216 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:01.769605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9fb9eb-081a-48fe-b7d8-1d7551c8bfb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:01.770487 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9fb9eb-081a-48fe-b7d8-1d7551c8bfb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:01.770768 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9fb9eb-081a-48fe-b7d8-1d7551c8bfb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:01.771057 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9fb9eb-081a-48fe-b7d8-1d7551c8bfb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:01.771728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9fb9eb-081a-48fe-b7d8-1d7551c8bfb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:01.773397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9fb9eb-081a-48fe-b7d8-1d7551c8bfb7 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:01.787275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 951/1925] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:01 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:01.789934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abfd8b3-d0b6-484b-b9fa-87a3bba59efc admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:01.801607 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abfd8b3-d0b6-484b-b9fa-87a3bba59efc admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:01.807220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72427036-a539-48c8-a884-76d7735de26f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:01.808324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72427036-a539-48c8-a884-76d7735de26f admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:01.808757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72427036-a539-48c8-a884-76d7735de26f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:01.809196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72427036-a539-48c8-a884-76d7735de26f admin admin] PATH_INFO: `/v3/projects/7812c61205d04a20806bf3f687a417d3/users/8a60e207de8c4a118fbf28a7b64ddcb4/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:01.828403 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72427036-a539-48c8-a884-76d7735de26f admin admin] RBAC: Authorizing `identity:create_grant(project_id=7812c61205d04a20806bf3f687a417d3, user_id=8a60e207de8c4a118fbf28a7b64ddcb4, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:01.831252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72427036-a539-48c8-a884-76d7735de26f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:01.950592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 975/1926] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:01 2022] PUT /identity/v3/projects/7812c61205d04a20806bf3f687a417d3/users/8a60e207de8c4a118fbf28a7b64ddcb4/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:01.962284 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3beeeb7-5189-4f48-bdae-259ce1b480fa None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:01.962564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3beeeb7-5189-4f48-bdae-259ce1b480fa None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:01.962834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3beeeb7-5189-4f48-bdae-259ce1b480fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:01.993383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d3beeeb7-5189-4f48-bdae-259ce1b480fa None None] MFA Rules not processed for user `8a60e207de8c4a118fbf28a7b64ddcb4`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:01.994614 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d3beeeb7-5189-4f48-bdae-259ce1b480fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:02.066872 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 952/1927] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:01 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:02.071269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72427036-a539-48c8-a884-76d7735de26f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:02.139420 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72427036-a539-48c8-a884-76d7735de26f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:02.145264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407911ce-5126-48d7-966d-164ceabdffe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:02.146102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407911ce-5126-48d7-966d-164ceabdffe3 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:02.146398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407911ce-5126-48d7-966d-164ceabdffe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:02.146667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407911ce-5126-48d7-966d-164ceabdffe3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:02.147306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407911ce-5126-48d7-966d-164ceabdffe3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:02.149232 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407911ce-5126-48d7-966d-164ceabdffe3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:02.299223 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 976/1928] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:02 2022] POST /identity/v3/projects => generated 385 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:50:02.300451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3beeeb7-5189-4f48-bdae-259ce1b480fa None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:02.312402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3beeeb7-5189-4f48-bdae-259ce1b480fa None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:02.367861 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:02.377898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d6ffc5-728e-4e5b-aece-726c11525626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:02.378597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d6ffc5-728e-4e5b-aece-726c11525626 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:02.378887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d6ffc5-728e-4e5b-aece-726c11525626 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:02.379282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d6ffc5-728e-4e5b-aece-726c11525626 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:02.379798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d6ffc5-728e-4e5b-aece-726c11525626 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:02.381985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d6ffc5-728e-4e5b-aece-726c11525626 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:02.485278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 953/1929] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:02 2022] POST /identity/v3/users => generated 299 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:50:02.486695 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407911ce-5126-48d7-966d-164ceabdffe3 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:02.499010 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407911ce-5126-48d7-966d-164ceabdffe3 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:02.504663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb70e98d-a473-4973-b64c-f8927574753c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:02.505095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb70e98d-a473-4973-b64c-f8927574753c admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:02.505382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb70e98d-a473-4973-b64c-f8927574753c admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:02.505658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb70e98d-a473-4973-b64c-f8927574753c admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:02.506430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb70e98d-a473-4973-b64c-f8927574753c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:02.507994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb70e98d-a473-4973-b64c-f8927574753c admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:02.523911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 977/1930] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:02 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:02.525198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d6ffc5-728e-4e5b-aece-726c11525626 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:02.549443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d6ffc5-728e-4e5b-aece-726c11525626 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:02.555304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a33012-8976-486d-a5d9-dc3c82f3eb1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:02.556395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a33012-8976-486d-a5d9-dc3c82f3eb1a admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:02.556892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a33012-8976-486d-a5d9-dc3c82f3eb1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:02.557376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a33012-8976-486d-a5d9-dc3c82f3eb1a admin admin] PATH_INFO: `/v3/projects/024468d73fd448ae9d9abd1cd8bfe5e6/users/09b922b93d0f4d80a493ab885a5dc096/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:02.580826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a33012-8976-486d-a5d9-dc3c82f3eb1a admin admin] RBAC: Authorizing `identity:create_grant(project_id=024468d73fd448ae9d9abd1cd8bfe5e6, user_id=09b922b93d0f4d80a493ab885a5dc096, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:02.587496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a33012-8976-486d-a5d9-dc3c82f3eb1a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:02.702108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 954/1931] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:02 2022] PUT /identity/v3/projects/024468d73fd448ae9d9abd1cd8bfe5e6/users/09b922b93d0f4d80a493ab885a5dc096/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:02.706341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb70e98d-a473-4973-b64c-f8927574753c admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:02.717759 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb70e98d-a473-4973-b64c-f8927574753c admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:02.768290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:02.783706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5410aa7-f515-42b5-8aba-e27b20204acd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:02.784368 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5410aa7-f515-42b5-8aba-e27b20204acd admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:02.784647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5410aa7-f515-42b5-8aba-e27b20204acd admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:02.784996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5410aa7-f515-42b5-8aba-e27b20204acd admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:02.785664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5410aa7-f515-42b5-8aba-e27b20204acd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:02.787330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5410aa7-f515-42b5-8aba-e27b20204acd admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:02.798932 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 978/1932] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:02 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:02.804319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a33012-8976-486d-a5d9-dc3c82f3eb1a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:02.815742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a33012-8976-486d-a5d9-dc3c82f3eb1a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:02.821969 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f700696-b318-4025-8889-e67865f854c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:02.822696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f700696-b318-4025-8889-e67865f854c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:02.822980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f700696-b318-4025-8889-e67865f854c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:02.823263 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f700696-b318-4025-8889-e67865f854c6 admin admin] PATH_INFO: `/v3/projects/024468d73fd448ae9d9abd1cd8bfe5e6/users/09b922b93d0f4d80a493ab885a5dc096/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:02.825262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f700696-b318-4025-8889-e67865f854c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=024468d73fd448ae9d9abd1cd8bfe5e6, user_id=09b922b93d0f4d80a493ab885a5dc096, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:02.828207 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f700696-b318-4025-8889-e67865f854c6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:02.839699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 955/1933] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:02 2022] PUT /identity/v3/projects/024468d73fd448ae9d9abd1cd8bfe5e6/users/09b922b93d0f4d80a493ab885a5dc096/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:02.851818 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c660fe8d-1c8b-44c2-87ef-2a9895cbfcac None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:02.852093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c660fe8d-1c8b-44c2-87ef-2a9895cbfcac None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:02.852373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c660fe8d-1c8b-44c2-87ef-2a9895cbfcac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:02.886175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c660fe8d-1c8b-44c2-87ef-2a9895cbfcac None None] MFA Rules not processed for user `09b922b93d0f4d80a493ab885a5dc096`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:02.887594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c660fe8d-1c8b-44c2-87ef-2a9895cbfcac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:02.978953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 979/1934] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:02 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:02.991901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f700696-b318-4025-8889-e67865f854c6 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:02.993749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6f700696-b318-4025-8889-e67865f854c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:03.060404 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f700696-b318-4025-8889-e67865f854c6 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:03.064152 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4774f8-55ff-4384-a6b9-b90b6c11d688 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:03.064808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4774f8-55ff-4384-a6b9-b90b6c11d688 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:03.065109 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4774f8-55ff-4384-a6b9-b90b6c11d688 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:03.065406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4774f8-55ff-4384-a6b9-b90b6c11d688 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:03.077731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4774f8-55ff-4384-a6b9-b90b6c11d688 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:03.077950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4774f8-55ff-4384-a6b9-b90b6c11d688 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:03.089447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 956/1935] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:02 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:50:03.196069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c660fe8d-1c8b-44c2-87ef-2a9895cbfcac None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:03.267905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c660fe8d-1c8b-44c2-87ef-2a9895cbfcac None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:03.272594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:03.273237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:03.273520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:03.273795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] PATH_INFO: `/v3/users/8a60e207de8c4a118fbf28a7b64ddcb4` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:03.275577 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a60e207de8c4a118fbf28a7b64ddcb4)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:03.277003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:03.528452 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a60e207de8c4a118fbf28a7b64ddcb4'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:03.529079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a60e207de8c4a118fbf28a7b64ddcb4'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:03.539888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a60e207de8c4a118fbf28a7b64ddcb4'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:03.540911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a60e207de8c4a118fbf28a7b64ddcb4'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:03.681276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a60e207de8c4a118fbf28a7b64ddcb4'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:03.689716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:03.708724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 980/1936] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:03 2022] DELETE /identity/v3/users/8a60e207de8c4a118fbf28a7b64ddcb4 => generated 0 bytes in 513 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:03.713054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47b6c09-0123-477a-bf3d-8618eebade65 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:03.713576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47b6c09-0123-477a-bf3d-8618eebade65 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:03.713688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47b6c09-0123-477a-bf3d-8618eebade65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:03.734783 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-c47b6c09-0123-477a-bf3d-8618eebade65 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:03.736045 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c47b6c09-0123-477a-bf3d-8618eebade65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:03.815253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 957/1937] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:03.826311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:03.838734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:03.897120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ec24e3-788e-4ce6-b35f-e7cf1723ff57 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:03.911831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b812b4-18b6-4f13-bcd5-51f112a0da9e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:03.911831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b812b4-18b6-4f13-bcd5-51f112a0da9e service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:03.911831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b812b4-18b6-4f13-bcd5-51f112a0da9e service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:03.911831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b812b4-18b6-4f13-bcd5-51f112a0da9e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:03.923437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b812b4-18b6-4f13-bcd5-51f112a0da9e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:03.924172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b812b4-18b6-4f13-bcd5-51f112a0da9e service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:03.940686 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 981/1938] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:04.575884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47b6c09-0123-477a-bf3d-8618eebade65 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:04.584040 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c47b6c09-0123-477a-bf3d-8618eebade65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:04.592410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47b6c09-0123-477a-bf3d-8618eebade65 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:04.598377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:04.599119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:04.599613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:04.599894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] PATH_INFO: `/v3/projects/7812c61205d04a20806bf3f687a417d3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:04.601075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7812c61205d04a20806bf3f687a417d3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:04.603032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:04.810896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:04.819460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7812c61205d04a20806bf3f687a417d3', 'request_id': 'req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:04.825092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7812c61205d04a20806bf3f687a417d3', 'request_id': 'req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:04.831319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7812c61205d04a20806bf3f687a417d3', 'request_id': 'req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:04.962185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] The token cache is being invalidate because project 7812c61205d04a20806bf3f687a417d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:04.962185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:04.964909 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:04.965418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 958/1939] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:04 2022] DELETE /identity/v3/projects/7812c61205d04a20806bf3f687a417d3 => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:04.969989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b812b4-18b6-4f13-bcd5-51f112a0da9e service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:04.983791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b812b4-18b6-4f13-bcd5-51f112a0da9e service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:05.026370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:05.029713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:05.070130 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8002b7f-09da-4324-949c-db9deb6892b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:05.070920 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8002b7f-09da-4324-949c-db9deb6892b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:05.071225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8002b7f-09da-4324-949c-db9deb6892b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:05.071522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8002b7f-09da-4324-949c-db9deb6892b1 admin admin] PATH_INFO: `/v3/users/09b922b93d0f4d80a493ab885a5dc096` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:05.072851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8002b7f-09da-4324-949c-db9deb6892b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=09b922b93d0f4d80a493ab885a5dc096)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:05.074971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8002b7f-09da-4324-949c-db9deb6892b1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:05.160257 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1daf6c93-c24a-4bc2-b64f-d76f53dc02c3 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:05.165930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:05.166935 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:05.167407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:05.167827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] PATH_INFO: `/v3/users/3c2790a3529843a9b2fd085c8dfc3fff` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:05.169282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c2790a3529843a9b2fd085c8dfc3fff)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:05.171306 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:05.369943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b8002b7f-09da-4324-949c-db9deb6892b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '09b922b93d0f4d80a493ab885a5dc096'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:05.371040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b8002b7f-09da-4324-949c-db9deb6892b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '09b922b93d0f4d80a493ab885a5dc096'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:05.380759 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b8002b7f-09da-4324-949c-db9deb6892b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '09b922b93d0f4d80a493ab885a5dc096'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:05.381740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b8002b7f-09da-4324-949c-db9deb6892b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '09b922b93d0f4d80a493ab885a5dc096'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:05.490546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c2790a3529843a9b2fd085c8dfc3fff'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:05.491214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c2790a3529843a9b2fd085c8dfc3fff'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:05.498747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c2790a3529843a9b2fd085c8dfc3fff'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:05.505625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b8002b7f-09da-4324-949c-db9deb6892b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '09b922b93d0f4d80a493ab885a5dc096'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:05.520945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c2790a3529843a9b2fd085c8dfc3fff'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:05.538509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 982/1940] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:04 2022] DELETE /identity/v3/users/09b922b93d0f4d80a493ab885a5dc096 => generated 0 bytes in 569 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:05.631920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c2790a3529843a9b2fd085c8dfc3fff'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:05.643971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 959/1941] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:05 2022] DELETE /identity/v3/users/3c2790a3529843a9b2fd085c8dfc3fff => generated 0 bytes in 615 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:05.645824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25342c93-bf86-4c17-918f-8db8a7a77f77 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:05.646194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25342c93-bf86-4c17-918f-8db8a7a77f77 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:05.646498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25342c93-bf86-4c17-918f-8db8a7a77f77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:05.680580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-25342c93-bf86-4c17-918f-8db8a7a77f77 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:05.680580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-25342c93-bf86-4c17-918f-8db8a7a77f77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:05.680580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:05.759944 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 983/1942] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:05.770414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:05.772050 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:05.836413 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f8843d-2d43-417a-886a-d2f249fc46ed admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:05.842348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f065b79d-7190-4a76-8e16-fe230bb56e28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:05.843031 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f065b79d-7190-4a76-8e16-fe230bb56e28 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:05.843364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f065b79d-7190-4a76-8e16-fe230bb56e28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:05.843790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f065b79d-7190-4a76-8e16-fe230bb56e28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:05.855566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f065b79d-7190-4a76-8e16-fe230bb56e28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:05.857339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f065b79d-7190-4a76-8e16-fe230bb56e28 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:05.870976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 960/1943] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:05.940523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:06.500920 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25342c93-bf86-4c17-918f-8db8a7a77f77 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:06.502682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-25342c93-bf86-4c17-918f-8db8a7a77f77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:06.516360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25342c93-bf86-4c17-918f-8db8a7a77f77 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:06.533049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:06.534260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:06.535330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:06.535805 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] PATH_INFO: `/v3/projects/024468d73fd448ae9d9abd1cd8bfe5e6` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:06.537540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] RBAC: Authorizing `identity:delete_project(project_id=024468d73fd448ae9d9abd1cd8bfe5e6)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:06.547715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:06.712099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f065b79d-7190-4a76-8e16-fe230bb56e28 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:06.713517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f065b79d-7190-4a76-8e16-fe230bb56e28 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:06.731226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '024468d73fd448ae9d9abd1cd8bfe5e6', 'request_id': 'req-68a009e4-8e23-4af7-8337-5fb2e6a242af'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:06.745459 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '024468d73fd448ae9d9abd1cd8bfe5e6', 'request_id': 'req-68a009e4-8e23-4af7-8337-5fb2e6a242af'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:06.822899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f065b79d-7190-4a76-8e16-fe230bb56e28 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:06.826589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '024468d73fd448ae9d9abd1cd8bfe5e6', 'request_id': 'req-68a009e4-8e23-4af7-8337-5fb2e6a242af'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:06.844795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] The token cache is being invalidate because project 024468d73fd448ae9d9abd1cd8bfe5e6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:06.845554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:06.846456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:06.850837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 984/1944] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:06 2022] DELETE /identity/v3/projects/024468d73fd448ae9d9abd1cd8bfe5e6 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:06.851954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:06.855021 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:06.950929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e33b6f-007f-4225-94de-8c807bdcd585 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:06.951565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e33b6f-007f-4225-94de-8c807bdcd585 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:06.952116 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e33b6f-007f-4225-94de-8c807bdcd585 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:06.952595 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e33b6f-007f-4225-94de-8c807bdcd585 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:06.954970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-04e33b6f-007f-4225-94de-8c807bdcd585 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:07.007789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a009e4-8e23-4af7-8337-5fb2e6a242af admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:07.015033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:07.016189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.016493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.016822 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b admin admin] PATH_INFO: `/v3/projects/5905682d27bf40ae816e75626ce522f9` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.018065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b admin admin] RBAC: Authorizing `identity:delete_project(project_id=5905682d27bf40ae816e75626ce522f9)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:07.020356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:07.039031 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e33b6f-007f-4225-94de-8c807bdcd585 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:07.041321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e33b6f-007f-4225-94de-8c807bdcd585 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:07.057952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 961/1945] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:06 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:50:07.063889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f602698b-3a63-447e-be80-89dff02dca96 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.064226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f602698b-3a63-447e-be80-89dff02dca96 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.064506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f602698b-3a63-447e-be80-89dff02dca96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.089983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f602698b-3a63-447e-be80-89dff02dca96 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:07.095089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f602698b-3a63-447e-be80-89dff02dca96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:07.101106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 962/1946] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:07.110205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da049433-36be-487f-8fab-33a5cdeaf6fa None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.110780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da049433-36be-487f-8fab-33a5cdeaf6fa None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.111158 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da049433-36be-487f-8fab-33a5cdeaf6fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.134514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-da049433-36be-487f-8fab-33a5cdeaf6fa None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:07.136006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-da049433-36be-487f-8fab-33a5cdeaf6fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:07.144718 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 963/1947] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:07.150417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da049433-36be-487f-8fab-33a5cdeaf6fa None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:07.160947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da049433-36be-487f-8fab-33a5cdeaf6fa None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:07.166519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac2b4ac-2629-47f0-98bf-8b314da7b6e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:07.167162 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac2b4ac-2629-47f0-98bf-8b314da7b6e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.167421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac2b4ac-2629-47f0-98bf-8b314da7b6e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.167919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac2b4ac-2629-47f0-98bf-8b314da7b6e4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.168813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac2b4ac-2629-47f0-98bf-8b314da7b6e4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:50:07.169190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac2b4ac-2629-47f0-98bf-8b314da7b6e4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:07.170951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac2b4ac-2629-47f0-98bf-8b314da7b6e4 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:07.185062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 964/1948] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:50:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:50:07.189223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac2b4ac-2629-47f0-98bf-8b314da7b6e4 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:07.201531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac2b4ac-2629-47f0-98bf-8b314da7b6e4 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:07.207696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ebf5ee-b749-471b-99b2-88069aa8b687 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:07.208472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ebf5ee-b749-471b-99b2-88069aa8b687 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.208757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ebf5ee-b749-471b-99b2-88069aa8b687 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.209188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ebf5ee-b749-471b-99b2-88069aa8b687 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.209839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ebf5ee-b749-471b-99b2-88069aa8b687 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:07.212358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ebf5ee-b749-471b-99b2-88069aa8b687 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:07.225580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:07.242346 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5905682d27bf40ae816e75626ce522f9', 'request_id': 'req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:07.248666 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5905682d27bf40ae816e75626ce522f9', 'request_id': 'req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:07.349755 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 965/1949] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:07 2022] POST /identity/v3/projects => generated 375 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:50:07.350944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ebf5ee-b749-471b-99b2-88069aa8b687 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:07.362410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:07.430649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5905682d27bf40ae816e75626ce522f9', 'request_id': 'req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:07.445562 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b admin admin] The token cache is being invalidate because project 5905682d27bf40ae816e75626ce522f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:07.446631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:07.447706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f7fa5d2a-0fb3-4f0f-9723-cbefb1ad185b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:07.449533 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 985/1950] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:06 2022] DELETE /identity/v3/projects/5905682d27bf40ae816e75626ce522f9 => generated 0 bytes in 598 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:07.456093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cc1eb4-7cae-4317-b184-38d28621d13c None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.456909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cc1eb4-7cae-4317-b184-38d28621d13c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.457429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cc1eb4-7cae-4317-b184-38d28621d13c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.483063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ebf5ee-b749-471b-99b2-88069aa8b687 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:07.484300 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c0cc1eb4-7cae-4317-b184-38d28621d13c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:07.485821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c0cc1eb4-7cae-4317-b184-38d28621d13c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:07.489358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cd8144-4fcf-4a08-9706-a6537c499761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:07.490208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cd8144-4fcf-4a08-9706-a6537c499761 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.490467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cd8144-4fcf-4a08-9706-a6537c499761 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.490738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cd8144-4fcf-4a08-9706-a6537c499761 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.491373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05cd8144-4fcf-4a08-9706-a6537c499761 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:07.493367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05cd8144-4fcf-4a08-9706-a6537c499761 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:07.496526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 986/1951] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:07.505869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04a66cb-5c59-46fb-bd85-919b1652db12 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.506679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04a66cb-5c59-46fb-bd85-919b1652db12 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.507340 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04a66cb-5c59-46fb-bd85-919b1652db12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.531370 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e04a66cb-5c59-46fb-bd85-919b1652db12 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:07.533028 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e04a66cb-5c59-46fb-bd85-919b1652db12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:07.550597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 987/1952] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:07.563959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04a66cb-5c59-46fb-bd85-919b1652db12 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:07.583198 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04a66cb-5c59-46fb-bd85-919b1652db12 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:07.588459 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ea8094-3268-44a9-8d74-d3ec96ff14a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:07.589153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ea8094-3268-44a9-8d74-d3ec96ff14a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.589529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ea8094-3268-44a9-8d74-d3ec96ff14a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.589892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ea8094-3268-44a9-8d74-d3ec96ff14a3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.590741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ea8094-3268-44a9-8d74-d3ec96ff14a3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:50:07.591100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ea8094-3268-44a9-8d74-d3ec96ff14a3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:07.592733 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ea8094-3268-44a9-8d74-d3ec96ff14a3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:07.609264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 988/1953] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:50:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:50:07.621436 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ea8094-3268-44a9-8d74-d3ec96ff14a3 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:07.621436 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ea8094-3268-44a9-8d74-d3ec96ff14a3 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:07.626124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab33e92-f273-4d2f-a025-35a907282881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:07.626796 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab33e92-f273-4d2f-a025-35a907282881 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.627070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab33e92-f273-4d2f-a025-35a907282881 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.627337 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab33e92-f273-4d2f-a025-35a907282881 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.627968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab33e92-f273-4d2f-a025-35a907282881 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:07.629969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab33e92-f273-4d2f-a025-35a907282881 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:07.653393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 966/1954] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:07 2022] POST /identity/v3/users => generated 288 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:07.659827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cd8144-4fcf-4a08-9706-a6537c499761 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:07.662419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-05cd8144-4fcf-4a08-9706-a6537c499761 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:07.674478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cd8144-4fcf-4a08-9706-a6537c499761 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:07.679954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c98a22-1766-42cc-8680-429d8c28b7cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:07.680696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c98a22-1766-42cc-8680-429d8c28b7cf admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.681198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c98a22-1766-42cc-8680-429d8c28b7cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.681379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c98a22-1766-42cc-8680-429d8c28b7cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.682094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c98a22-1766-42cc-8680-429d8c28b7cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:07.683732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c98a22-1766-42cc-8680-429d8c28b7cf admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:07.695347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 967/1955] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:07 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:50:07.700221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b98093d-3c3b-43bd-9219-4af2ef74d4d1 None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.700985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b98093d-3c3b-43bd-9219-4af2ef74d4d1 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.701404 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b98093d-3c3b-43bd-9219-4af2ef74d4d1 None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.703075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 968/1956] 10.222.0.65 () {54 vars in 921 bytes} [Wed Aug 24 03:50:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 24 03:50:07.704037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b98093d-3c3b-43bd-9219-4af2ef74d4d1 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:07.745313 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b98093d-3c3b-43bd-9219-4af2ef74d4d1 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:07.750786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb968b23-db3f-41b9-8754-f198ecb9444c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:07.751677 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb968b23-db3f-41b9-8754-f198ecb9444c admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.751953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb968b23-db3f-41b9-8754-f198ecb9444c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.752220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb968b23-db3f-41b9-8754-f198ecb9444c admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.752892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb968b23-db3f-41b9-8754-f198ecb9444c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:07.755315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb968b23-db3f-41b9-8754-f198ecb9444c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:07.766898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 969/1957] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:07 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:07.768476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ac7c58-91a5-44af-a180-61412f8f6d48 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.768752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ac7c58-91a5-44af-a180-61412f8f6d48 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.769401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ac7c58-91a5-44af-a180-61412f8f6d48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.790218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d8ac7c58-91a5-44af-a180-61412f8f6d48 None None] MFA Rules not processed for user `4b347c9c89a64168b51c439b4a92d647`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:07.791433 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d8ac7c58-91a5-44af-a180-61412f8f6d48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:07.806746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 989/1958] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:07 2022] POST /identity/v3/projects => generated 375 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:07.806746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab33e92-f273-4d2f-a025-35a907282881 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:07.867734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 970/1959] 10.222.0.65 () {58 vars in 1018 bytes} [Wed Aug 24 03:50:07 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:07.868923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ac7c58-91a5-44af-a180-61412f8f6d48 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:07.881531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab33e92-f273-4d2f-a025-35a907282881 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:07.881749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ac7c58-91a5-44af-a180-61412f8f6d48 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:07.886240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8915fe9d-8ca1-437f-94ff-7be7846b2a86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:07.886470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527b7c1e-f123-47b7-b0e7-d032b99f6df3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:07.886894 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8915fe9d-8ca1-437f-94ff-7be7846b2a86 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.887155 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527b7c1e-f123-47b7-b0e7-d032b99f6df3 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:07.887249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8915fe9d-8ca1-437f-94ff-7be7846b2a86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.887427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8915fe9d-8ca1-437f-94ff-7be7846b2a86 admin admin] PATH_INFO: `/v3/projects/225668676bfd4193abb66b91754f25ae/users/52edd210c55c43c28dd832dd0c36c883/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.887519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527b7c1e-f123-47b7-b0e7-d032b99f6df3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:07.887691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527b7c1e-f123-47b7-b0e7-d032b99f6df3 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:07.888323 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527b7c1e-f123-47b7-b0e7-d032b99f6df3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:07.890320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527b7c1e-f123-47b7-b0e7-d032b99f6df3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:07.904207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8915fe9d-8ca1-437f-94ff-7be7846b2a86 admin admin] RBAC: Authorizing `identity:create_grant(project_id=225668676bfd4193abb66b91754f25ae, user_id=52edd210c55c43c28dd832dd0c36c883, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:07.907013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8915fe9d-8ca1-437f-94ff-7be7846b2a86 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:07.985799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 990/1960] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:07 2022] PUT /identity/v3/projects/225668676bfd4193abb66b91754f25ae/users/52edd210c55c43c28dd832dd0c36c883/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:07.987002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8915fe9d-8ca1-437f-94ff-7be7846b2a86 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:07.989334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8915fe9d-8ca1-437f-94ff-7be7846b2a86 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:08.008820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 971/1961] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:07 2022] POST /identity/v3/users => generated 288 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:08.008820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527b7c1e-f123-47b7-b0e7-d032b99f6df3 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:08.010869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-527b7c1e-f123-47b7-b0e7-d032b99f6df3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:08.026208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527b7c1e-f123-47b7-b0e7-d032b99f6df3 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:08.084617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8915fe9d-8ca1-437f-94ff-7be7846b2a86 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:08.090166 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13304377-bf76-45e8-a5a3-0871995bba02 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:08.090793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13304377-bf76-45e8-a5a3-0871995bba02 service placement] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:08.091069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13304377-bf76-45e8-a5a3-0871995bba02 service placement] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:08.091335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13304377-bf76-45e8-a5a3-0871995bba02 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:08.095651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:08.096290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:08.096571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:08.098952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] PATH_INFO: `/v3/users/728c3f2735d6441b870cf146fbf084c0` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:08.098952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] RBAC: Authorizing `identity:delete_user(user_id=728c3f2735d6441b870cf146fbf084c0)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:08.100058 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:08.101138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13304377-bf76-45e8-a5a3-0871995bba02 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:08.102842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13304377-bf76-45e8-a5a3-0871995bba02 service placement] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:08.114126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 991/1962] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:08.115714 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30726915-a08e-467b-9aa9-04b85113254f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:08.115997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30726915-a08e-467b-9aa9-04b85113254f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:08.116268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30726915-a08e-467b-9aa9-04b85113254f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:08.149938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-30726915-a08e-467b-9aa9-04b85113254f None None] MFA Rules not processed for user `52edd210c55c43c28dd832dd0c36c883`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:08.151259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-30726915-a08e-467b-9aa9-04b85113254f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:08.177849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:08.227835 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 992/1963] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:08 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:08.229448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30726915-a08e-467b-9aa9-04b85113254f None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:08.242903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30726915-a08e-467b-9aa9-04b85113254f None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:08.247588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713370b1-ec2d-48bd-83d7-0ea9bedd4227 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:08.248181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713370b1-ec2d-48bd-83d7-0ea9bedd4227 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:08.248599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713370b1-ec2d-48bd-83d7-0ea9bedd4227 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:08.248881 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713370b1-ec2d-48bd-83d7-0ea9bedd4227 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:08.249536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713370b1-ec2d-48bd-83d7-0ea9bedd4227 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:08.251183 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713370b1-ec2d-48bd-83d7-0ea9bedd4227 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:08.262259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 993/1964] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:08 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:08.263473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713370b1-ec2d-48bd-83d7-0ea9bedd4227 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:08.273978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713370b1-ec2d-48bd-83d7-0ea9bedd4227 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:08.278755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb12eee2-4b65-4c30-a6d6-7d9d36c956ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:08.279371 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb12eee2-4b65-4c30-a6d6-7d9d36c956ac admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:08.279638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb12eee2-4b65-4c30-a6d6-7d9d36c956ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:08.279912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb12eee2-4b65-4c30-a6d6-7d9d36c956ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:08.280545 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb12eee2-4b65-4c30-a6d6-7d9d36c956ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:08.282482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb12eee2-4b65-4c30-a6d6-7d9d36c956ac admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:08.311755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:DELETE=5,keystone:SELECT=92,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:08.327870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '728c3f2735d6441b870cf146fbf084c0'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:08.328539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '728c3f2735d6441b870cf146fbf084c0'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:08.336628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '728c3f2735d6441b870cf146fbf084c0'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:08.348662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '728c3f2735d6441b870cf146fbf084c0'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:08.371501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 994/1965] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:08 2022] POST /identity/v3/projects => generated 373 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:08.372787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb12eee2-4b65-4c30-a6d6-7d9d36c956ac admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:08.458903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb12eee2-4b65-4c30-a6d6-7d9d36c956ac admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:08.464685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '728c3f2735d6441b870cf146fbf084c0'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:08.467230 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90197e35-925f-4b09-8c3f-06b00dbaa224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:08.467925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90197e35-925f-4b09-8c3f-06b00dbaa224 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:08.468141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90197e35-925f-4b09-8c3f-06b00dbaa224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:08.468409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90197e35-925f-4b09-8c3f-06b00dbaa224 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:08.469065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90197e35-925f-4b09-8c3f-06b00dbaa224 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:08.470726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90197e35-925f-4b09-8c3f-06b00dbaa224 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:08.476744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 972/1966] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:07 2022] DELETE /identity/v3/users/728c3f2735d6441b870cf146fbf084c0 => generated 0 bytes in 475 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:08.476744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:08.476744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:08.493246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 995/1967] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:08 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:08.509276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90197e35-925f-4b09-8c3f-06b00dbaa224 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:08.513013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-90197e35-925f-4b09-8c3f-06b00dbaa224 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:08.530852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90197e35-925f-4b09-8c3f-06b00dbaa224 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:08.555724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40826523-cc47-4131-b015-d95b6af5585e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:08.560735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c23ac68-a7e9-4e8f-b922-b8eb5aaad809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:08.561338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c23ac68-a7e9-4e8f-b922-b8eb5aaad809 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:08.562013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c23ac68-a7e9-4e8f-b922-b8eb5aaad809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:08.562013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c23ac68-a7e9-4e8f-b922-b8eb5aaad809 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:08.562531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c23ac68-a7e9-4e8f-b922-b8eb5aaad809 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:08.564451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c23ac68-a7e9-4e8f-b922-b8eb5aaad809 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:08.606812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c49208-aa80-4b70-8013-a8645a16f53c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:08.613767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c49208-aa80-4b70-8013-a8645a16f53c admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:08.613767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c49208-aa80-4b70-8013-a8645a16f53c admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:08.613767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c49208-aa80-4b70-8013-a8645a16f53c admin admin] PATH_INFO: `/v3/projects/8e330c99e9284df5b883750795b345ba/users/74db39b3ce974a30916dd58f3bc455a0/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:08.624104 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c49208-aa80-4b70-8013-a8645a16f53c admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e330c99e9284df5b883750795b345ba, user_id=74db39b3ce974a30916dd58f3bc455a0, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:08.626847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c49208-aa80-4b70-8013-a8645a16f53c admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:08.711674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 973/1968] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:08 2022] POST /identity/v3/users => generated 293 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:08.713330 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c23ac68-a7e9-4e8f-b922-b8eb5aaad809 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:08.724652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c23ac68-a7e9-4e8f-b922-b8eb5aaad809 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:08.731916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b12b29-838f-4010-9ddb-9d2622a2de4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:08.732553 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b12b29-838f-4010-9ddb-9d2622a2de4d admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:08.732835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b12b29-838f-4010-9ddb-9d2622a2de4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:08.733105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b12b29-838f-4010-9ddb-9d2622a2de4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:08.733774 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b12b29-838f-4010-9ddb-9d2622a2de4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:08.735427 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b12b29-838f-4010-9ddb-9d2622a2de4d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:08.747467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 974/1969] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:08 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:08.752394 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b12b29-838f-4010-9ddb-9d2622a2de4d admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:08.752823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 996/1970] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:08 2022] PUT /identity/v3/projects/8e330c99e9284df5b883750795b345ba/users/74db39b3ce974a30916dd58f3bc455a0/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:08.763872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c16aa1-f910-42fd-95bb-06413ebdbab6 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:08.764212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c16aa1-f910-42fd-95bb-06413ebdbab6 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:08.764612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c16aa1-f910-42fd-95bb-06413ebdbab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:08.772280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b12b29-838f-4010-9ddb-9d2622a2de4d admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:08.817180 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-b3c16aa1-f910-42fd-95bb-06413ebdbab6 None None] MFA Rules not processed for user `74db39b3ce974a30916dd58f3bc455a0`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:08.818444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b3c16aa1-f910-42fd-95bb-06413ebdbab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:08.824329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:08.839392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feeb09ca-73c2-44b3-a492-a9f7eab14b99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:08.840089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feeb09ca-73c2-44b3-a492-a9f7eab14b99 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:08.840518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feeb09ca-73c2-44b3-a492-a9f7eab14b99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:08.840790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feeb09ca-73c2-44b3-a492-a9f7eab14b99 admin admin] PATH_INFO: `/v3/projects/e73e7da3feb540049c99a6bed5d19333/users/7df77aa59e8847b583cf59b90795a1ee/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:08.864453 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feeb09ca-73c2-44b3-a492-a9f7eab14b99 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e73e7da3feb540049c99a6bed5d19333, user_id=7df77aa59e8847b583cf59b90795a1ee, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:08.868249 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feeb09ca-73c2-44b3-a492-a9f7eab14b99 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:08.908160 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 997/1971] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:08 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:08.909334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c16aa1-f910-42fd-95bb-06413ebdbab6 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:08.911336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b3c16aa1-f910-42fd-95bb-06413ebdbab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:08.926081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c16aa1-f910-42fd-95bb-06413ebdbab6 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:08.930960 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09a1342-0548-46b1-a151-83772d6dc798 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:08.931605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09a1342-0548-46b1-a151-83772d6dc798 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:08.931883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09a1342-0548-46b1-a151-83772d6dc798 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:08.932152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09a1342-0548-46b1-a151-83772d6dc798 admin admin] PATH_INFO: `/v3/projects/dacde99115e641439164fc91e75f2aa0` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:08.933328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09a1342-0548-46b1-a151-83772d6dc798 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dacde99115e641439164fc91e75f2aa0)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:08.935382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09a1342-0548-46b1-a151-83772d6dc798 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:08.969316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 975/1972] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:08 2022] PUT /identity/v3/projects/e73e7da3feb540049c99a6bed5d19333/users/7df77aa59e8847b583cf59b90795a1ee/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:08.970267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feeb09ca-73c2-44b3-a492-a9f7eab14b99 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:08.972009 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-feeb09ca-73c2-44b3-a492-a9f7eab14b99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:09.038723 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feeb09ca-73c2-44b3-a492-a9f7eab14b99 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:09.043056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033ac9f4-4c6b-4f66-81a1-3cf05eb7719e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:09.043838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033ac9f4-4c6b-4f66-81a1-3cf05eb7719e service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.044190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033ac9f4-4c6b-4f66-81a1-3cf05eb7719e service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.044518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033ac9f4-4c6b-4f66-81a1-3cf05eb7719e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.085376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f09a1342-0548-46b1-a151-83772d6dc798 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dacde99115e641439164fc91e75f2aa0', 'request_id': 'req-f09a1342-0548-46b1-a151-83772d6dc798'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:09.091798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f09a1342-0548-46b1-a151-83772d6dc798 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dacde99115e641439164fc91e75f2aa0', 'request_id': 'req-f09a1342-0548-46b1-a151-83772d6dc798'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:09.116132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033ac9f4-4c6b-4f66-81a1-3cf05eb7719e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:09.117630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033ac9f4-4c6b-4f66-81a1-3cf05eb7719e service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:09.131881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 976/1973] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:08 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:09.133536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033ac9f4-4c6b-4f66-81a1-3cf05eb7719e service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:09.172679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f09a1342-0548-46b1-a151-83772d6dc798 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dacde99115e641439164fc91e75f2aa0', 'request_id': 'req-f09a1342-0548-46b1-a151-83772d6dc798'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:09.176612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:09.189388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f09a1342-0548-46b1-a151-83772d6dc798 admin admin] The token cache is being invalidate because project dacde99115e641439164fc91e75f2aa0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:09.189676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f09a1342-0548-46b1-a151-83772d6dc798 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:09.190392 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f09a1342-0548-46b1-a151-83772d6dc798 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:09.193145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 998/1974] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:08 2022] DELETE /identity/v3/projects/dacde99115e641439164fc91e75f2aa0 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:09.197658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e064163c-fc82-43c6-bf89-d96138bd8069 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.197950 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e064163c-fc82-43c6-bf89-d96138bd8069 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.198335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e064163c-fc82-43c6-bf89-d96138bd8069 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.202794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033ac9f4-4c6b-4f66-81a1-3cf05eb7719e service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:09.220556 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e064163c-fc82-43c6-bf89-d96138bd8069 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:09.222773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e064163c-fc82-43c6-bf89-d96138bd8069 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:09.266418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd0d792-2806-4779-b4d9-8c1f627010a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:09.267036 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd0d792-2806-4779-b4d9-8c1f627010a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.267316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd0d792-2806-4779-b4d9-8c1f627010a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.267652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd0d792-2806-4779-b4d9-8c1f627010a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.268311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd0d792-2806-4779-b4d9-8c1f627010a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:09.270542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd0d792-2806-4779-b4d9-8c1f627010a7 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:09.281016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 977/1975] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:09 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:09.287848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd0d792-2806-4779-b4d9-8c1f627010a7 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:09.287848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ccd0d792-2806-4779-b4d9-8c1f627010a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:09.288491 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 999/1976] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:09.296007 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee06248-0f11-46f8-a876-f470807ec6e1 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.296226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee06248-0f11-46f8-a876-f470807ec6e1 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.296566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee06248-0f11-46f8-a876-f470807ec6e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.301868 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd0d792-2806-4779-b4d9-8c1f627010a7 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:09.306646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f072c1b-0fba-4a3c-a665-c447abef575d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:09.310567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f072c1b-0fba-4a3c-a665-c447abef575d admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.310567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f072c1b-0fba-4a3c-a665-c447abef575d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.310567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f072c1b-0fba-4a3c-a665-c447abef575d admin admin] PATH_INFO: `/v3/projects/e73e7da3feb540049c99a6bed5d19333/users/7df77aa59e8847b583cf59b90795a1ee/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.310567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f072c1b-0fba-4a3c-a665-c447abef575d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e73e7da3feb540049c99a6bed5d19333, user_id=7df77aa59e8847b583cf59b90795a1ee, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:09.312514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f072c1b-0fba-4a3c-a665-c447abef575d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:09.317068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-cee06248-0f11-46f8-a876-f470807ec6e1 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:09.318308 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-cee06248-0f11-46f8-a876-f470807ec6e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:09.324362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 978/1977] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:09 2022] PUT /identity/v3/projects/e73e7da3feb540049c99a6bed5d19333/users/7df77aa59e8847b583cf59b90795a1ee/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:09.336513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a0d135-d16b-4843-94dd-933c748e4377 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.336879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a0d135-d16b-4843-94dd-933c748e4377 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.337214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a0d135-d16b-4843-94dd-933c748e4377 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.366199 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-a3a0d135-d16b-4843-94dd-933c748e4377 None None] MFA Rules not processed for user `7df77aa59e8847b583cf59b90795a1ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:09.367486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a3a0d135-d16b-4843-94dd-933c748e4377 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:09.382711 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1000/1978] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:09.389103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee06248-0f11-46f8-a876-f470807ec6e1 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:09.401115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee06248-0f11-46f8-a876-f470807ec6e1 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:09.413151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750ec6e6-f1bf-4ed6-8f0c-e9292117b8ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:09.414206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750ec6e6-f1bf-4ed6-8f0c-e9292117b8ff admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.414497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750ec6e6-f1bf-4ed6-8f0c-e9292117b8ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.414783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750ec6e6-f1bf-4ed6-8f0c-e9292117b8ff admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.415466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750ec6e6-f1bf-4ed6-8f0c-e9292117b8ff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:50:09.415774 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750ec6e6-f1bf-4ed6-8f0c-e9292117b8ff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:09.418561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750ec6e6-f1bf-4ed6-8f0c-e9292117b8ff admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:09.431427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1001/1979] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:50:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:50:09.436822 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750ec6e6-f1bf-4ed6-8f0c-e9292117b8ff admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:09.446380 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750ec6e6-f1bf-4ed6-8f0c-e9292117b8ff admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:09.451144 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817e2425-6be9-42b1-a47a-97737cd27c7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:09.451749 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817e2425-6be9-42b1-a47a-97737cd27c7a admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.452031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817e2425-6be9-42b1-a47a-97737cd27c7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.452327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817e2425-6be9-42b1-a47a-97737cd27c7a admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.452942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817e2425-6be9-42b1-a47a-97737cd27c7a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:09.453054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 979/1980] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:09 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:09.454938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817e2425-6be9-42b1-a47a-97737cd27c7a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:09.465426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a0d135-d16b-4843-94dd-933c748e4377 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:09.467279 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a3a0d135-d16b-4843-94dd-933c748e4377 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:09.531896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a0d135-d16b-4843-94dd-933c748e4377 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:09.536020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed2a6ac-3d5a-4327-b20d-32fbc4937601 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:09.536628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed2a6ac-3d5a-4327-b20d-32fbc4937601 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.536924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed2a6ac-3d5a-4327-b20d-32fbc4937601 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.540229 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed2a6ac-3d5a-4327-b20d-32fbc4937601 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.540229 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3ed2a6ac-3d5a-4327-b20d-32fbc4937601 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:09.600002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed2a6ac-3d5a-4327-b20d-32fbc4937601 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:09.601536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed2a6ac-3d5a-4327-b20d-32fbc4937601 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:09.610169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1002/1981] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:09 2022] POST /identity/v3/projects => generated 401 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:50:09.610825 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 980/1982] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:09 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:50:09.615307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817e2425-6be9-42b1-a47a-97737cd27c7a admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:09.637412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:09.687829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817e2425-6be9-42b1-a47a-97737cd27c7a admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:09.693584 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6499ecfe-56f2-4b95-89bc-7ebebe5fe4b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:09.694057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6499ecfe-56f2-4b95-89bc-7ebebe5fe4b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.694366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6499ecfe-56f2-4b95-89bc-7ebebe5fe4b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.694637 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6499ecfe-56f2-4b95-89bc-7ebebe5fe4b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.695288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6499ecfe-56f2-4b95-89bc-7ebebe5fe4b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:09.697345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6499ecfe-56f2-4b95-89bc-7ebebe5fe4b0 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:09.763394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:09.810288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1003/1983] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:09 2022] POST /identity/v3/users => generated 301 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:50:09.814450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed2a6ac-3d5a-4327-b20d-32fbc4937601 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:09.826567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed2a6ac-3d5a-4327-b20d-32fbc4937601 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:09.832701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e55d62f-d984-4198-80a1-a8f3f85c4e3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:09.833377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e55d62f-d984-4198-80a1-a8f3f85c4e3a admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.833659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e55d62f-d984-4198-80a1-a8f3f85c4e3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.833926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e55d62f-d984-4198-80a1-a8f3f85c4e3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.834601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e55d62f-d984-4198-80a1-a8f3f85c4e3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:09.836194 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e55d62f-d984-4198-80a1-a8f3f85c4e3a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:09.847422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 981/1984] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:50:09.852016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6499ecfe-56f2-4b95-89bc-7ebebe5fe4b0 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:09.863160 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6499ecfe-56f2-4b95-89bc-7ebebe5fe4b0 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:09.868616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50986433-bcf0-4e66-bfbb-d67b65e076c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:09.869246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50986433-bcf0-4e66-bfbb-d67b65e076c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.869530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50986433-bcf0-4e66-bfbb-d67b65e076c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.869797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50986433-bcf0-4e66-bfbb-d67b65e076c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.870493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50986433-bcf0-4e66-bfbb-d67b65e076c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:09.872112 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50986433-bcf0-4e66-bfbb-d67b65e076c9 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:09.884318 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1004/1985] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:09.890594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e55d62f-d984-4198-80a1-a8f3f85c4e3a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:09.900828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e55d62f-d984-4198-80a1-a8f3f85c4e3a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:09.906598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7084b70-6d34-4cc5-9d62-b89b33dfc5a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:09.907431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7084b70-6d34-4cc5-9d62-b89b33dfc5a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:09.907787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7084b70-6d34-4cc5-9d62-b89b33dfc5a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:09.908145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7084b70-6d34-4cc5-9d62-b89b33dfc5a4 admin admin] PATH_INFO: `/v3/projects/e70fd9043c4345afab3059b080ae2b53/users/4c1c594d16dd455386e82236b0df50c7/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:09.936865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7084b70-6d34-4cc5-9d62-b89b33dfc5a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e70fd9043c4345afab3059b080ae2b53, user_id=4c1c594d16dd455386e82236b0df50c7, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:09.940583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7084b70-6d34-4cc5-9d62-b89b33dfc5a4 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:10.083216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 982/1986] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:09 2022] PUT /identity/v3/projects/e70fd9043c4345afab3059b080ae2b53/users/4c1c594d16dd455386e82236b0df50c7/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:10.092554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e2dda0-b6b9-4cae-a8d0-42da8d639af9 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:10.092840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e2dda0-b6b9-4cae-a8d0-42da8d639af9 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:10.093112 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e2dda0-b6b9-4cae-a8d0-42da8d639af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:10.126164 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-78e2dda0-b6b9-4cae-a8d0-42da8d639af9 None None] MFA Rules not processed for user `4c1c594d16dd455386e82236b0df50c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:10.127425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-78e2dda0-b6b9-4cae-a8d0-42da8d639af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:10.208946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1005/1987] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:10 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:10.213775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7084b70-6d34-4cc5-9d62-b89b33dfc5a4 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:10.280255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7084b70-6d34-4cc5-9d62-b89b33dfc5a4 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:10.285936 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b237b9-e042-4317-8618-b9955e9435b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:10.286666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b237b9-e042-4317-8618-b9955e9435b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:10.286947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b237b9-e042-4317-8618-b9955e9435b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:10.287218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b237b9-e042-4317-8618-b9955e9435b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:10.287848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b237b9-e042-4317-8618-b9955e9435b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:10.289768 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b237b9-e042-4317-8618-b9955e9435b2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:10.439285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 983/1988] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:10 2022] POST /identity/v3/projects => generated 403 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:10.443912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e2dda0-b6b9-4cae-a8d0-42da8d639af9 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:10.455354 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e2dda0-b6b9-4cae-a8d0-42da8d639af9 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:10.521792 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60cff48-1dc2-4f7f-812e-401c9db741f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:10.522566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60cff48-1dc2-4f7f-812e-401c9db741f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:10.522843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60cff48-1dc2-4f7f-812e-401c9db741f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:10.523428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60cff48-1dc2-4f7f-812e-401c9db741f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:10.527183 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60cff48-1dc2-4f7f-812e-401c9db741f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:10.529237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60cff48-1dc2-4f7f-812e-401c9db741f5 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:10.608971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:10.665376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1006/1989] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:10 2022] POST /identity/v3/users => generated 308 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:10.669297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b237b9-e042-4317-8618-b9955e9435b2 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:10.684926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b237b9-e042-4317-8618-b9955e9435b2 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:10.691092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f47c313-6c94-430e-bc8e-96663ec2cbd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:10.692371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f47c313-6c94-430e-bc8e-96663ec2cbd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:10.692729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f47c313-6c94-430e-bc8e-96663ec2cbd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:10.693083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f47c313-6c94-430e-bc8e-96663ec2cbd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:10.693825 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f47c313-6c94-430e-bc8e-96663ec2cbd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:10.695552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f47c313-6c94-430e-bc8e-96663ec2cbd6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:10.709027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 984/1990] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:10 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:10.713513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60cff48-1dc2-4f7f-812e-401c9db741f5 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:10.724255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60cff48-1dc2-4f7f-812e-401c9db741f5 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:10.729156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996af0d0-e1ae-4bff-8129-6523feff87c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:10.729831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996af0d0-e1ae-4bff-8129-6523feff87c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:10.730108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996af0d0-e1ae-4bff-8129-6523feff87c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:10.730412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996af0d0-e1ae-4bff-8129-6523feff87c6 admin admin] PATH_INFO: `/v3/projects/c771ba557cba4f288fd0a678bd2c5f95/users/c6223806aff847ea9b0a40d61f0a6d07/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:10.747138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996af0d0-e1ae-4bff-8129-6523feff87c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c771ba557cba4f288fd0a678bd2c5f95, user_id=c6223806aff847ea9b0a40d61f0a6d07, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:10.749912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996af0d0-e1ae-4bff-8129-6523feff87c6 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:10.928474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1007/1991] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:10 2022] PUT /identity/v3/projects/c771ba557cba4f288fd0a678bd2c5f95/users/c6223806aff847ea9b0a40d61f0a6d07/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:10.929806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f47c313-6c94-430e-bc8e-96663ec2cbd6 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:10.939953 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f47c313-6c94-430e-bc8e-96663ec2cbd6 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:10.999148 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88517c7-43e1-47e4-95ed-48387488204d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:10.999764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88517c7-43e1-47e4-95ed-48387488204d admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:11.000049 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88517c7-43e1-47e4-95ed-48387488204d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:11.000366 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88517c7-43e1-47e4-95ed-48387488204d admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:11.001022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88517c7-43e1-47e4-95ed-48387488204d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:11.003196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88517c7-43e1-47e4-95ed-48387488204d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:11.016471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 985/1992] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:10 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:11.021220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996af0d0-e1ae-4bff-8129-6523feff87c6 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:11.032425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996af0d0-e1ae-4bff-8129-6523feff87c6 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:11.046439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0777bd5-53df-4e5f-876f-f00dbe13cc17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:11.047398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0777bd5-53df-4e5f-876f-f00dbe13cc17 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:11.047886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0777bd5-53df-4e5f-876f-f00dbe13cc17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:11.048318 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0777bd5-53df-4e5f-876f-f00dbe13cc17 admin admin] PATH_INFO: `/v3/projects/c771ba557cba4f288fd0a678bd2c5f95/users/c6223806aff847ea9b0a40d61f0a6d07/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:11.064286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0777bd5-53df-4e5f-876f-f00dbe13cc17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c771ba557cba4f288fd0a678bd2c5f95, user_id=c6223806aff847ea9b0a40d61f0a6d07, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:11.067544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0777bd5-53df-4e5f-876f-f00dbe13cc17 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:11.097524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1008/1993] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:11 2022] PUT /identity/v3/projects/c771ba557cba4f288fd0a678bd2c5f95/users/c6223806aff847ea9b0a40d61f0a6d07/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:11.108288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b8ee93-22ae-4857-a772-baa3afecd453 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:11.108776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b8ee93-22ae-4857-a772-baa3afecd453 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:11.108906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b8ee93-22ae-4857-a772-baa3afecd453 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:11.140229 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-51b8ee93-22ae-4857-a772-baa3afecd453 None None] MFA Rules not processed for user `c6223806aff847ea9b0a40d61f0a6d07`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:11.142374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-51b8ee93-22ae-4857-a772-baa3afecd453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:11.175132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:11.218320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 986/1994] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:11 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:11.232509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0777bd5-53df-4e5f-876f-f00dbe13cc17 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:11.234267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f0777bd5-53df-4e5f-876f-f00dbe13cc17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:11.294696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0777bd5-53df-4e5f-876f-f00dbe13cc17 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:11.298517 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703ec6e0-2c96-4174-a240-0c41a2ea8f12 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:11.299130 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703ec6e0-2c96-4174-a240-0c41a2ea8f12 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:11.299407 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703ec6e0-2c96-4174-a240-0c41a2ea8f12 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:11.299676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703ec6e0-2c96-4174-a240-0c41a2ea8f12 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:11.309809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703ec6e0-2c96-4174-a240-0c41a2ea8f12 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:11.311308 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703ec6e0-2c96-4174-a240-0c41a2ea8f12 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:11.322021 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1009/1995] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:11 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:11.328836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:11.722773 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b8ee93-22ae-4857-a772-baa3afecd453 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:11.791342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b8ee93-22ae-4857-a772-baa3afecd453 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:11.796010 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:11.796659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:11.796933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:11.797196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] PATH_INFO: `/v3/users/4c1c594d16dd455386e82236b0df50c7` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:11.798345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4c1c594d16dd455386e82236b0df50c7)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:11.801188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:12.072302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4c1c594d16dd455386e82236b0df50c7'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:12.072986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4c1c594d16dd455386e82236b0df50c7'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:12.079851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4c1c594d16dd455386e82236b0df50c7'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:12.091879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4c1c594d16dd455386e82236b0df50c7'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:12.185262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4c1c594d16dd455386e82236b0df50c7'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:12.192700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 987/1996] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:11 2022] DELETE /identity/v3/users/4c1c594d16dd455386e82236b0df50c7 => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:12.197394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6317084-af30-4504-82eb-173b05ff82f2 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:12.197940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6317084-af30-4504-82eb-173b05ff82f2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:12.198177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6317084-af30-4504-82eb-173b05ff82f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:12.209218 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:12.219930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d6317084-af30-4504-82eb-173b05ff82f2 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:12.221141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d6317084-af30-4504-82eb-173b05ff82f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:12.282793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1010/1997] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:12.294530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:12.295966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:12.342772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:12.357625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05faf06-846d-4eed-9bb9-aca576f16981 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:12.363460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc823fa-e4a6-4d45-8433-0c602685a511 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:12.363460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc823fa-e4a6-4d45-8433-0c602685a511 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:12.363460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc823fa-e4a6-4d45-8433-0c602685a511 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:12.363460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc823fa-e4a6-4d45-8433-0c602685a511 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:12.372962 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc823fa-e4a6-4d45-8433-0c602685a511 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:12.374433 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc823fa-e4a6-4d45-8433-0c602685a511 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:12.384331 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 988/1998] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:50:12.958818 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6317084-af30-4504-82eb-173b05ff82f2 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:12.960355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d6317084-af30-4504-82eb-173b05ff82f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:12.976478 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6317084-af30-4504-82eb-173b05ff82f2 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:12.981464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:12.982258 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:12.982406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:12.982673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] PATH_INFO: `/v3/projects/e70fd9043c4345afab3059b080ae2b53` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:12.984006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e70fd9043c4345afab3059b080ae2b53)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:12.986288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:13.212281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e70fd9043c4345afab3059b080ae2b53', 'request_id': 'req-91d5e426-7be4-402f-9f4c-16710e74549e'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:13.218631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e70fd9043c4345afab3059b080ae2b53', 'request_id': 'req-91d5e426-7be4-402f-9f4c-16710e74549e'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:13.315093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e70fd9043c4345afab3059b080ae2b53', 'request_id': 'req-91d5e426-7be4-402f-9f4c-16710e74549e'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:13.332474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] The token cache is being invalidate because project e70fd9043c4345afab3059b080ae2b53 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:13.341157 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:13.341157 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:13.341157 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1011/1999] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:12 2022] DELETE /identity/v3/projects/e70fd9043c4345afab3059b080ae2b53 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:13.341412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc823fa-e4a6-4d45-8433-0c602685a511 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:13.351142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc823fa-e4a6-4d45-8433-0c602685a511 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:13.416440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:13.416440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:13.416440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:13.416440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] PATH_INFO: `/v3/users/c6223806aff847ea9b0a40d61f0a6d07` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:13.417683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6223806aff847ea9b0a40d61f0a6d07)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:13.419699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:13.760694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6223806aff847ea9b0a40d61f0a6d07'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:13.761262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6223806aff847ea9b0a40d61f0a6d07'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:13.768286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6223806aff847ea9b0a40d61f0a6d07'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:13.783506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6223806aff847ea9b0a40d61f0a6d07'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:13.865413 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6223806aff847ea9b0a40d61f0a6d07'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:13.873041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 989/2000] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:13 2022] DELETE /identity/v3/users/c6223806aff847ea9b0a40d61f0a6d07 => generated 0 bytes in 534 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:14.211713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:14.213220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:14.227372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d5e426-7be4-402f-9f4c-16710e74549e admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:14.295775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1f291b-54b6-4400-9462-503ef95f413c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:14.296492 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1f291b-54b6-4400-9462-503ef95f413c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:14.296772 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1f291b-54b6-4400-9462-503ef95f413c admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:14.297048 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1f291b-54b6-4400-9462-503ef95f413c admin admin] PATH_INFO: `/v3/users/52edd210c55c43c28dd832dd0c36c883` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:14.298375 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1f291b-54b6-4400-9462-503ef95f413c admin admin] RBAC: Authorizing `identity:delete_user(user_id=52edd210c55c43c28dd832dd0c36c883)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:14.301234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1f291b-54b6-4400-9462-503ef95f413c admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:14.351877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:14.353690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:14.371608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692caf88-f499-41ea-9b87-357d2e988045 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:14.377265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350ad849-2f9f-4e93-9cfd-34a01a705b26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:14.383999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350ad849-2f9f-4e93-9cfd-34a01a705b26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:14.383999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350ad849-2f9f-4e93-9cfd-34a01a705b26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:14.383999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350ad849-2f9f-4e93-9cfd-34a01a705b26 admin admin] PATH_INFO: `/v3/projects/c771ba557cba4f288fd0a678bd2c5f95` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:14.383999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350ad849-2f9f-4e93-9cfd-34a01a705b26 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c771ba557cba4f288fd0a678bd2c5f95)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:14.383999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350ad849-2f9f-4e93-9cfd-34a01a705b26 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:14.523796 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:14.538355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8f1f291b-54b6-4400-9462-503ef95f413c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52edd210c55c43c28dd832dd0c36c883'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:14.538919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-350ad849-2f9f-4e93-9cfd-34a01a705b26 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c771ba557cba4f288fd0a678bd2c5f95', 'request_id': 'req-350ad849-2f9f-4e93-9cfd-34a01a705b26'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:14.539118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8f1f291b-54b6-4400-9462-503ef95f413c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52edd210c55c43c28dd832dd0c36c883'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:14.543961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:14.545347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-350ad849-2f9f-4e93-9cfd-34a01a705b26 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c771ba557cba4f288fd0a678bd2c5f95', 'request_id': 'req-350ad849-2f9f-4e93-9cfd-34a01a705b26'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:14.546861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8f1f291b-54b6-4400-9462-503ef95f413c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52edd210c55c43c28dd832dd0c36c883'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:14.547830 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8f1f291b-54b6-4400-9462-503ef95f413c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52edd210c55c43c28dd832dd0c36c883'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:14.552603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-350ad849-2f9f-4e93-9cfd-34a01a705b26 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c771ba557cba4f288fd0a678bd2c5f95', 'request_id': 'req-350ad849-2f9f-4e93-9cfd-34a01a705b26'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:14.661845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8f1f291b-54b6-4400-9462-503ef95f413c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52edd210c55c43c28dd832dd0c36c883'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:14.690882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1012/2001] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:14 2022] DELETE /identity/v3/users/52edd210c55c43c28dd832dd0c36c883 => generated 0 bytes in 479 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:14.695574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498c516d-6e8d-4ece-a690-1dbfd71601d4 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:14.696457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498c516d-6e8d-4ece-a690-1dbfd71601d4 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:14.696898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498c516d-6e8d-4ece-a690-1dbfd71601d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:14.718244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-498c516d-6e8d-4ece-a690-1dbfd71601d4 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:14.719518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-498c516d-6e8d-4ece-a690-1dbfd71601d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:14.786074 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-350ad849-2f9f-4e93-9cfd-34a01a705b26 admin admin] The token cache is being invalidate because project c771ba557cba4f288fd0a678bd2c5f95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:14.786074 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-350ad849-2f9f-4e93-9cfd-34a01a705b26 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:14.786074 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-350ad849-2f9f-4e93-9cfd-34a01a705b26 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:14.788383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 990/2002] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:14 2022] DELETE /identity/v3/projects/c771ba557cba4f288fd0a678bd2c5f95 => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:14.792789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1013/2003] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:14.793148 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27230ddf-9201-4bc3-a313-13fffdf361fa None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:14.793483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27230ddf-9201-4bc3-a313-13fffdf361fa None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:14.793830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27230ddf-9201-4bc3-a313-13fffdf361fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:14.804358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498c516d-6e8d-4ece-a690-1dbfd71601d4 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:14.805836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-498c516d-6e8d-4ece-a690-1dbfd71601d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:14.820594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-27230ddf-9201-4bc3-a313-13fffdf361fa None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:14.822637 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-27230ddf-9201-4bc3-a313-13fffdf361fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:14.831797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 991/2004] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:14.839373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf599da-38e2-4e1b-b766-685b3d657e90 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:14.839839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf599da-38e2-4e1b-b766-685b3d657e90 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:14.840176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf599da-38e2-4e1b-b766-685b3d657e90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:14.862051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-7bf599da-38e2-4e1b-b766-685b3d657e90 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:14.863728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7bf599da-38e2-4e1b-b766-685b3d657e90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:14.870025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498c516d-6e8d-4ece-a690-1dbfd71601d4 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:14.880899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 992/2005] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:14.884511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29728f6-cc28-4b24-b91d-64abf8dad864 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:14.885134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29728f6-cc28-4b24-b91d-64abf8dad864 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:14.885410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29728f6-cc28-4b24-b91d-64abf8dad864 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:14.885684 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29728f6-cc28-4b24-b91d-64abf8dad864 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:14.887523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf599da-38e2-4e1b-b766-685b3d657e90 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:14.887667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a29728f6-cc28-4b24-b91d-64abf8dad864 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:14.897978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf599da-38e2-4e1b-b766-685b3d657e90 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:14.902808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd35949e-f7e7-4b98-b4d7-3dec3e710922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:14.903503 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd35949e-f7e7-4b98-b4d7-3dec3e710922 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:14.903627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29728f6-cc28-4b24-b91d-64abf8dad864 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:14.903748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd35949e-f7e7-4b98-b4d7-3dec3e710922 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:14.903967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd35949e-f7e7-4b98-b4d7-3dec3e710922 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:14.904632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd35949e-f7e7-4b98-b4d7-3dec3e710922 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:50:14.904931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd35949e-f7e7-4b98-b4d7-3dec3e710922 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:14.905091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29728f6-cc28-4b24-b91d-64abf8dad864 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:14.909915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd35949e-f7e7-4b98-b4d7-3dec3e710922 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:14.921329 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1014/2006] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:50:14.922455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 993/2007] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:50:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:50:14.927727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29728f6-cc28-4b24-b91d-64abf8dad864 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:14.938334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29728f6-cc28-4b24-b91d-64abf8dad864 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:14.945133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a369f37b-71f3-4bad-9118-20ab92098a93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:14.951352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a369f37b-71f3-4bad-9118-20ab92098a93 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:14.951352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a369f37b-71f3-4bad-9118-20ab92098a93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:14.951352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a369f37b-71f3-4bad-9118-20ab92098a93 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:14.951352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a369f37b-71f3-4bad-9118-20ab92098a93 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:14.951352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a369f37b-71f3-4bad-9118-20ab92098a93 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:15.038425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1015/2008] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:14 2022] POST /identity/v3/projects => generated 405 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:15.043131 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd35949e-f7e7-4b98-b4d7-3dec3e710922 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:15.054998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd35949e-f7e7-4b98-b4d7-3dec3e710922 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:15.116301 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58173bd6-d71f-4e4a-836d-ba077f149794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:15.116917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58173bd6-d71f-4e4a-836d-ba077f149794 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:15.117196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58173bd6-d71f-4e4a-836d-ba077f149794 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:15.117470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58173bd6-d71f-4e4a-836d-ba077f149794 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:15.118100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58173bd6-d71f-4e4a-836d-ba077f149794 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:15.120170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58173bd6-d71f-4e4a-836d-ba077f149794 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:15.257659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 994/2009] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:15 2022] POST /identity/v3/users => generated 303 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:15.259866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a369f37b-71f3-4bad-9118-20ab92098a93 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:15.270677 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a369f37b-71f3-4bad-9118-20ab92098a93 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:15.275484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3efabf-eb01-4aba-896f-00d2a4d0b0c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:15.276093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3efabf-eb01-4aba-896f-00d2a4d0b0c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:15.276551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3efabf-eb01-4aba-896f-00d2a4d0b0c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:15.276850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3efabf-eb01-4aba-896f-00d2a4d0b0c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:15.277499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3efabf-eb01-4aba-896f-00d2a4d0b0c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:15.279146 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3efabf-eb01-4aba-896f-00d2a4d0b0c9 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:15.290928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1016/2010] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:15.301266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58173bd6-d71f-4e4a-836d-ba077f149794 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:15.317596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58173bd6-d71f-4e4a-836d-ba077f149794 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:15.320196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3efabf-eb01-4aba-896f-00d2a4d0b0c9 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:15.322027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7e3efabf-eb01-4aba-896f-00d2a4d0b0c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:15.324894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4491d7-476a-4801-a17f-67cdf12ea0b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:15.325857 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4491d7-476a-4801-a17f-67cdf12ea0b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:15.326350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4491d7-476a-4801-a17f-67cdf12ea0b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:15.326798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4491d7-476a-4801-a17f-67cdf12ea0b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:15.327831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4491d7-476a-4801-a17f-67cdf12ea0b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:15.329904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4491d7-476a-4801-a17f-67cdf12ea0b2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:15.336518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3efabf-eb01-4aba-896f-00d2a4d0b0c9 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:15.341929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf358ef-7c87-47a8-a960-e7d079120542 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:15.348799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf358ef-7c87-47a8-a960-e7d079120542 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:15.348799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf358ef-7c87-47a8-a960-e7d079120542 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:15.348799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf358ef-7c87-47a8-a960-e7d079120542 admin admin] PATH_INFO: `/v3/users/74db39b3ce974a30916dd58f3bc455a0` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:15.348799 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf358ef-7c87-47a8-a960-e7d079120542 admin admin] RBAC: Authorizing `identity:delete_user(user_id=74db39b3ce974a30916dd58f3bc455a0)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:15.349103 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 995/2011] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:15 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:15.349193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf358ef-7c87-47a8-a960-e7d079120542 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:15.350470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4491d7-476a-4801-a17f-67cdf12ea0b2 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:15.361564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=92,keystone:DELETE=6,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:15.365909 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4491d7-476a-4801-a17f-67cdf12ea0b2 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:15.371071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec92656-f987-4e48-a139-0614dc393f35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:15.371729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec92656-f987-4e48-a139-0614dc393f35 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:15.372009 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec92656-f987-4e48-a139-0614dc393f35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:15.372289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec92656-f987-4e48-a139-0614dc393f35 admin admin] PATH_INFO: `/v3/projects/806e734529f54d569ef68b63587f704f/users/abad1328adbb484c9478fa8e2c1817af/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:15.388386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec92656-f987-4e48-a139-0614dc393f35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=806e734529f54d569ef68b63587f704f, user_id=abad1328adbb484c9478fa8e2c1817af, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:15.391124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec92656-f987-4e48-a139-0614dc393f35 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:15.538858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 996/2012] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:15 2022] PUT /identity/v3/projects/806e734529f54d569ef68b63587f704f/users/abad1328adbb484c9478fa8e2c1817af/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:15.539705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec92656-f987-4e48-a139-0614dc393f35 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:15.541989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3ec92656-f987-4e48-a139-0614dc393f35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:15.580183 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-fcf358ef-7c87-47a8-a960-e7d079120542 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74db39b3ce974a30916dd58f3bc455a0'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:15.580858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-fcf358ef-7c87-47a8-a960-e7d079120542 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74db39b3ce974a30916dd58f3bc455a0'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:15.588148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-fcf358ef-7c87-47a8-a960-e7d079120542 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74db39b3ce974a30916dd58f3bc455a0'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:15.589065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-fcf358ef-7c87-47a8-a960-e7d079120542 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74db39b3ce974a30916dd58f3bc455a0'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:15.612851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec92656-f987-4e48-a139-0614dc393f35 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:15.617983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:15.618658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:15.618934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:15.619227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] PATH_INFO: `/v3/projects/225668676bfd4193abb66b91754f25ae` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:15.620344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] RBAC: Authorizing `identity:delete_project(project_id=225668676bfd4193abb66b91754f25ae)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:15.622308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:15.639167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:15.745453 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-fcf358ef-7c87-47a8-a960-e7d079120542 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74db39b3ce974a30916dd58f3bc455a0'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:15.765156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1017/2013] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:15 2022] DELETE /identity/v3/users/74db39b3ce974a30916dd58f3bc455a0 => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:15.767049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee40a23-3a85-457a-ac31-776653d64409 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:15.767707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee40a23-3a85-457a-ac31-776653d64409 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:15.767868 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee40a23-3a85-457a-ac31-776653d64409 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:15.791499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-9ee40a23-3a85-457a-ac31-776653d64409 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:15.792820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9ee40a23-3a85-457a-ac31-776653d64409 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:15.801352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '225668676bfd4193abb66b91754f25ae', 'request_id': 'req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:15.807598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '225668676bfd4193abb66b91754f25ae', 'request_id': 'req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:15.816640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '225668676bfd4193abb66b91754f25ae', 'request_id': 'req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:15.857675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1018/2014] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:15.858993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575dfedc-0edc-4db8-8262-d22c3b7fcf04 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:15.859313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575dfedc-0edc-4db8-8262-d22c3b7fcf04 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:15.859585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575dfedc-0edc-4db8-8262-d22c3b7fcf04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:15.882013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-575dfedc-0edc-4db8-8262-d22c3b7fcf04 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:15.883422 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-575dfedc-0edc-4db8-8262-d22c3b7fcf04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:15.891500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1019/2015] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:15.892455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575dfedc-0edc-4db8-8262-d22c3b7fcf04 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:15.894008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-575dfedc-0edc-4db8-8262-d22c3b7fcf04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:15.947101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] The token cache is being invalidate because project 225668676bfd4193abb66b91754f25ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:15.947481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:15.948106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:15.950007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 997/2016] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:15 2022] DELETE /identity/v3/projects/225668676bfd4193abb66b91754f25ae => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:15.950999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:15.952403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:15.968227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575dfedc-0edc-4db8-8262-d22c3b7fcf04 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:15.973850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3879fb4-747f-4800-aeb3-8a6c0ef7c2c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:15.974513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3879fb4-747f-4800-aeb3-8a6c0ef7c2c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:15.974794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3879fb4-747f-4800-aeb3-8a6c0ef7c2c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:15.975069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3879fb4-747f-4800-aeb3-8a6c0ef7c2c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:15.976932 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e3879fb4-747f-4800-aeb3-8a6c0ef7c2c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:15.996041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3879fb4-747f-4800-aeb3-8a6c0ef7c2c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:15.997586 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3879fb4-747f-4800-aeb3-8a6c0ef7c2c1 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:16.009768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1020/2017] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:16.014788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3879fb4-747f-4800-aeb3-8a6c0ef7c2c1 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:16.014788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e3879fb4-747f-4800-aeb3-8a6c0ef7c2c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:16.024051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3879fb4-747f-4800-aeb3-8a6c0ef7c2c1 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:16.028980 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addd9e30-6b97-4677-bed7-2d50449edc1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:16.029625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd9e30-6b97-4677-bed7-2d50449edc1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:16.029903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd9e30-6b97-4677-bed7-2d50449edc1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:16.030226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addd9e30-6b97-4677-bed7-2d50449edc1c admin admin] PATH_INFO: `/v3/users/7df77aa59e8847b583cf59b90795a1ee` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:16.031493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addd9e30-6b97-4677-bed7-2d50449edc1c admin admin] RBAC: Authorizing `identity:delete_user(user_id=7df77aa59e8847b583cf59b90795a1ee)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:16.033355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addd9e30-6b97-4677-bed7-2d50449edc1c admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:16.037524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff93ccc-dd30-44ee-861d-1b20f4d2e74b admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:16.049268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aacb730-349c-442b-b494-72c2874f4afe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:16.049268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aacb730-349c-442b-b494-72c2874f4afe service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:16.049268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aacb730-349c-442b-b494-72c2874f4afe service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:16.049268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aacb730-349c-442b-b494-72c2874f4afe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:16.050547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-8aacb730-349c-442b-b494-72c2874f4afe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:16.064818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aacb730-349c-442b-b494-72c2874f4afe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:16.066345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aacb730-349c-442b-b494-72c2874f4afe service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:16.076328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 998/2018] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:16.077137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aacb730-349c-442b-b494-72c2874f4afe service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:16.079895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-8aacb730-349c-442b-b494-72c2874f4afe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:16.090434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aacb730-349c-442b-b494-72c2874f4afe service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:16.095270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:16.095895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:16.096174 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:16.096448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] PATH_INFO: `/v3/users/17c69ff6f9314c3387461af42eba52f8` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:16.098334 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=17c69ff6f9314c3387461af42eba52f8)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:16.100372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:16.298466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-addd9e30-6b97-4677-bed7-2d50449edc1c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7df77aa59e8847b583cf59b90795a1ee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:16.299323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-addd9e30-6b97-4677-bed7-2d50449edc1c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7df77aa59e8847b583cf59b90795a1ee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:16.307159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-addd9e30-6b97-4677-bed7-2d50449edc1c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7df77aa59e8847b583cf59b90795a1ee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:16.307836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-addd9e30-6b97-4677-bed7-2d50449edc1c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7df77aa59e8847b583cf59b90795a1ee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:16.382915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '17c69ff6f9314c3387461af42eba52f8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:16.384626 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '17c69ff6f9314c3387461af42eba52f8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:16.394636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '17c69ff6f9314c3387461af42eba52f8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:16.396942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-addd9e30-6b97-4677-bed7-2d50449edc1c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7df77aa59e8847b583cf59b90795a1ee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:16.399841 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:16.420893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '17c69ff6f9314c3387461af42eba52f8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:16.428866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1021/2019] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:16 2022] DELETE /identity/v3/users/7df77aa59e8847b583cf59b90795a1ee => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:16.578328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '17c69ff6f9314c3387461af42eba52f8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:16.585327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 999/2020] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:16 2022] DELETE /identity/v3/users/17c69ff6f9314c3387461af42eba52f8 => generated 0 bytes in 508 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:16.590053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeddc526-afb4-495d-ad42-3392ce250e4f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:16.590376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeddc526-afb4-495d-ad42-3392ce250e4f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:16.590657 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeddc526-afb4-495d-ad42-3392ce250e4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:16.611368 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-eeddc526-afb4-495d-ad42-3392ce250e4f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:16.612661 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-eeddc526-afb4-495d-ad42-3392ce250e4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:16.673956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1022/2021] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:16.681215 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:16.682781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:16.684369 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeddc526-afb4-495d-ad42-3392ce250e4f None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:16.685973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-eeddc526-afb4-495d-ad42-3392ce250e4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:16.698036 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24526a63-3ced-44ef-af80-e58595ddba1a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:16.703362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c316bb-defa-487e-8500-a3b8a37edb25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:16.704012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c316bb-defa-487e-8500-a3b8a37edb25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:16.704460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c316bb-defa-487e-8500-a3b8a37edb25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:16.704758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c316bb-defa-487e-8500-a3b8a37edb25 admin admin] PATH_INFO: `/v3/projects/8e330c99e9284df5b883750795b345ba` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:16.706746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c316bb-defa-487e-8500-a3b8a37edb25 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e330c99e9284df5b883750795b345ba)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:16.713752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c316bb-defa-487e-8500-a3b8a37edb25 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:16.726422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:16.764683 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeddc526-afb4-495d-ad42-3392ce250e4f None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:16.772559 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436fca1b-14ce-4ee3-8e99-5a5ae1dd3be7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:16.773776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436fca1b-14ce-4ee3-8e99-5a5ae1dd3be7 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:16.774065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436fca1b-14ce-4ee3-8e99-5a5ae1dd3be7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:16.774384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436fca1b-14ce-4ee3-8e99-5a5ae1dd3be7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:16.787007 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436fca1b-14ce-4ee3-8e99-5a5ae1dd3be7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:16.788578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436fca1b-14ce-4ee3-8e99-5a5ae1dd3be7 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:16.802243 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1023/2022] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:50:16.887665 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-33c316bb-defa-487e-8500-a3b8a37edb25 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e330c99e9284df5b883750795b345ba', 'request_id': 'req-33c316bb-defa-487e-8500-a3b8a37edb25'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:16.899245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-33c316bb-defa-487e-8500-a3b8a37edb25 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e330c99e9284df5b883750795b345ba', 'request_id': 'req-33c316bb-defa-487e-8500-a3b8a37edb25'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:16.906403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-33c316bb-defa-487e-8500-a3b8a37edb25 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e330c99e9284df5b883750795b345ba', 'request_id': 'req-33c316bb-defa-487e-8500-a3b8a37edb25'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.031676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-33c316bb-defa-487e-8500-a3b8a37edb25 admin admin] The token cache is being invalidate because project 8e330c99e9284df5b883750795b345ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:17.033510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436fca1b-14ce-4ee3-8e99-5a5ae1dd3be7 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:17.034406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-33c316bb-defa-487e-8500-a3b8a37edb25 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.037879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-436fca1b-14ce-4ee3-8e99-5a5ae1dd3be7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:17.040650 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-33c316bb-defa-487e-8500-a3b8a37edb25 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.049143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1000/2023] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:16 2022] DELETE /identity/v3/projects/8e330c99e9284df5b883750795b345ba => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:17.051920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce94fd79-8351-475d-ae56-8c036218dc38 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.052287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce94fd79-8351-475d-ae56-8c036218dc38 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.052576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce94fd79-8351-475d-ae56-8c036218dc38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.059039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436fca1b-14ce-4ee3-8e99-5a5ae1dd3be7 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:17.115233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-ce94fd79-8351-475d-ae56-8c036218dc38 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:17.116495 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ce94fd79-8351-475d-ae56-8c036218dc38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:17.202765 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dde7d1-3a28-418e-a95c-700b326a0627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:17.203507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dde7d1-3a28-418e-a95c-700b326a0627 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.203858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dde7d1-3a28-418e-a95c-700b326a0627 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.204220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dde7d1-3a28-418e-a95c-700b326a0627 admin admin] PATH_INFO: `/v3/projects/e73e7da3feb540049c99a6bed5d19333` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.207345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dde7d1-3a28-418e-a95c-700b326a0627 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e73e7da3feb540049c99a6bed5d19333)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:17.207857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dde7d1-3a28-418e-a95c-700b326a0627 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:17.242225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1001/2024] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:17.250601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582d1a5c-8c5e-4ba6-920a-35a2e2169e7a None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.251106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582d1a5c-8c5e-4ba6-920a-35a2e2169e7a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.251612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582d1a5c-8c5e-4ba6-920a-35a2e2169e7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.281292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-582d1a5c-8c5e-4ba6-920a-35a2e2169e7a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:17.282559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-582d1a5c-8c5e-4ba6-920a-35a2e2169e7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:17.292311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1002/2025] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:17.296748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582d1a5c-8c5e-4ba6-920a-35a2e2169e7a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:17.307922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582d1a5c-8c5e-4ba6-920a-35a2e2169e7a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:17.316389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d341f265-8a09-44ad-b6eb-d09132dd887e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:17.320265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d341f265-8a09-44ad-b6eb-d09132dd887e admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.321960 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d341f265-8a09-44ad-b6eb-d09132dd887e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.322898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d341f265-8a09-44ad-b6eb-d09132dd887e admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.327074 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d341f265-8a09-44ad-b6eb-d09132dd887e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:50:17.327074 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d341f265-8a09-44ad-b6eb-d09132dd887e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:17.329197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d341f265-8a09-44ad-b6eb-d09132dd887e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:17.353395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1003/2026] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:50:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:50:17.357202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f5dde7d1-3a28-418e-a95c-700b326a0627 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e73e7da3feb540049c99a6bed5d19333', 'request_id': 'req-f5dde7d1-3a28-418e-a95c-700b326a0627'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.359253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d341f265-8a09-44ad-b6eb-d09132dd887e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:17.366266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f5dde7d1-3a28-418e-a95c-700b326a0627 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e73e7da3feb540049c99a6bed5d19333', 'request_id': 'req-f5dde7d1-3a28-418e-a95c-700b326a0627'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.375867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d341f265-8a09-44ad-b6eb-d09132dd887e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:17.448236 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ee750d-44e8-4605-9e55-d4783a525887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:17.448838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ee750d-44e8-4605-9e55-d4783a525887 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.449098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ee750d-44e8-4605-9e55-d4783a525887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.449506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ee750d-44e8-4605-9e55-d4783a525887 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.450238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ee750d-44e8-4605-9e55-d4783a525887 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:17.452257 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ee750d-44e8-4605-9e55-d4783a525887 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:17.476529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f5dde7d1-3a28-418e-a95c-700b326a0627 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e73e7da3feb540049c99a6bed5d19333', 'request_id': 'req-f5dde7d1-3a28-418e-a95c-700b326a0627'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.489395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f5dde7d1-3a28-418e-a95c-700b326a0627 admin admin] The token cache is being invalidate because project e73e7da3feb540049c99a6bed5d19333 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:17.489712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f5dde7d1-3a28-418e-a95c-700b326a0627 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.490624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f5dde7d1-3a28-418e-a95c-700b326a0627 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.492353 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1024/2027] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:17 2022] DELETE /identity/v3/projects/e73e7da3feb540049c99a6bed5d19333 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:17.497261 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2169978d-5a2d-4978-8a8e-26ea5ca62371 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.497538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2169978d-5a2d-4978-8a8e-26ea5ca62371 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.498027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2169978d-5a2d-4978-8a8e-26ea5ca62371 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.524271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-2169978d-5a2d-4978-8a8e-26ea5ca62371 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:17.525802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-2169978d-5a2d-4978-8a8e-26ea5ca62371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:17.534802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1025/2028] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:17.543033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d34f1e-9d52-4ffb-bb35-07d5359e4d21 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.543943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d34f1e-9d52-4ffb-bb35-07d5359e4d21 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.543943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d34f1e-9d52-4ffb-bb35-07d5359e4d21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.556034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1004/2029] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:17 2022] POST /identity/v3/projects => generated 383 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:17.557338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ee750d-44e8-4605-9e55-d4783a525887 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:17.559498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-17ee750d-44e8-4605-9e55-d4783a525887 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:17.568880 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-91d34f1e-9d52-4ffb-bb35-07d5359e4d21 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:17.571050 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-91d34f1e-9d52-4ffb-bb35-07d5359e4d21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:17.646679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1026/2030] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:17.647776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d34f1e-9d52-4ffb-bb35-07d5359e4d21 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:17.649262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-91d34f1e-9d52-4ffb-bb35-07d5359e4d21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:17.658862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ee750d-44e8-4605-9e55-d4783a525887 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:17.662733 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d34f1e-9d52-4ffb-bb35-07d5359e4d21 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:17.664696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1117dc-a3bb-4457-a7e7-36e1f7ac9c47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:17.665316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1117dc-a3bb-4457-a7e7-36e1f7ac9c47 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.665596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1117dc-a3bb-4457-a7e7-36e1f7ac9c47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.665938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1117dc-a3bb-4457-a7e7-36e1f7ac9c47 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.666586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1117dc-a3bb-4457-a7e7-36e1f7ac9c47 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:17.667649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:17.668290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.668501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1117dc-a3bb-4457-a7e7-36e1f7ac9c47 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:17.668606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.668828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] PATH_INFO: `/v3/projects/7585c97131fa4697abfa45e1d002d1b9` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.670215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7585c97131fa4697abfa45e1d002d1b9)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:17.671926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:17.809794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1005/2031] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:17 2022] POST /identity/v3/users => generated 292 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:17.810698 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1117dc-a3bb-4457-a7e7-36e1f7ac9c47 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:17.820898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1117dc-a3bb-4457-a7e7-36e1f7ac9c47 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:17.821481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7585c97131fa4697abfa45e1d002d1b9', 'request_id': 'req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.827184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7585c97131fa4697abfa45e1d002d1b9', 'request_id': 'req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.886557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba628fc-c0fe-4880-8943-a00940fab5a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:17.887178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba628fc-c0fe-4880-8943-a00940fab5a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.887453 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba628fc-c0fe-4880-8943-a00940fab5a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.887722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba628fc-c0fe-4880-8943-a00940fab5a2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.888381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba628fc-c0fe-4880-8943-a00940fab5a2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:50:17.888668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba628fc-c0fe-4880-8943-a00940fab5a2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:17.890294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba628fc-c0fe-4880-8943-a00940fab5a2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:17.906679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1006/2032] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:50:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:50:17.912642 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba628fc-c0fe-4880-8943-a00940fab5a2 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:17.927998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba628fc-c0fe-4880-8943-a00940fab5a2 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:17.933466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7585c97131fa4697abfa45e1d002d1b9', 'request_id': 'req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.950813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0adabc-556b-4d78-94a7-22de422ba303 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:17.950813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0adabc-556b-4d78-94a7-22de422ba303 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.950813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0adabc-556b-4d78-94a7-22de422ba303 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.950813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0adabc-556b-4d78-94a7-22de422ba303 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.951255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0adabc-556b-4d78-94a7-22de422ba303 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:17.951255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0adabc-556b-4d78-94a7-22de422ba303 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:17.954141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] The token cache is being invalidate because project 7585c97131fa4697abfa45e1d002d1b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:17.954465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.955528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:17.958871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1027/2033] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:17 2022] DELETE /identity/v3/projects/7585c97131fa4697abfa45e1d002d1b9 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:17.960249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:17.963372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:17.968345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1007/2034] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:17 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:17.968345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5002330a-97df-48cd-8118-7144872607b1 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.968345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5002330a-97df-48cd-8118-7144872607b1 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.968896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5002330a-97df-48cd-8118-7144872607b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.980968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53dee59-4c17-43bd-b5be-2bce5f7a5e93 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:17.989756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e555015-f0a0-46b9-838e-f90da98475b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:17.990823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e555015-f0a0-46b9-838e-f90da98475b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:17.991244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e555015-f0a0-46b9-838e-f90da98475b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:17.991570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e555015-f0a0-46b9-838e-f90da98475b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:17.992338 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e555015-f0a0-46b9-838e-f90da98475b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:17.996224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e555015-f0a0-46b9-838e-f90da98475b3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:18.011924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-5002330a-97df-48cd-8118-7144872607b1 None None] MFA Rules not processed for user `abad1328adbb484c9478fa8e2c1817af`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:18.013738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5002330a-97df-48cd-8118-7144872607b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:18.091845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1008/2035] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:17 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:18.091845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5002330a-97df-48cd-8118-7144872607b1 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:18.091845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5002330a-97df-48cd-8118-7144872607b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:18.105897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5002330a-97df-48cd-8118-7144872607b1 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:18.110735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:18.111380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:18.111658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:18.111930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] PATH_INFO: `/v3/users/b189506adf4c41409cf878fe7df0bb91` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:18.113978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b189506adf4c41409cf878fe7df0bb91)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:18.115943 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:18.130506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1028/2036] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:17 2022] POST /identity/v3/projects => generated 375 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:18.135499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e555015-f0a0-46b9-838e-f90da98475b3 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:18.135499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4e555015-f0a0-46b9-838e-f90da98475b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:18.201948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e555015-f0a0-46b9-838e-f90da98475b3 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:18.207043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07761bb-e749-41e7-ae3b-51c959957e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:18.207654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07761bb-e749-41e7-ae3b-51c959957e71 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:18.207929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07761bb-e749-41e7-ae3b-51c959957e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:18.208200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07761bb-e749-41e7-ae3b-51c959957e71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:18.209960 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07761bb-e749-41e7-ae3b-51c959957e71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:18.210497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07761bb-e749-41e7-ae3b-51c959957e71 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:18.221084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1029/2037] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:18 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:18.222668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07761bb-e749-41e7-ae3b-51c959957e71 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:18.224949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c07761bb-e749-41e7-ae3b-51c959957e71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:18.239463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07761bb-e749-41e7-ae3b-51c959957e71 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:18.244270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f04e8c-721b-4831-8097-31443e5b34e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:18.244875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f04e8c-721b-4831-8097-31443e5b34e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:18.245150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f04e8c-721b-4831-8097-31443e5b34e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:18.245417 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f04e8c-721b-4831-8097-31443e5b34e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:18.246065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f04e8c-721b-4831-8097-31443e5b34e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:18.248085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f04e8c-721b-4831-8097-31443e5b34e7 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:18.260342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:18.438558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1030/2038] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:18 2022] POST /identity/v3/projects => generated 403 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:18.439565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f04e8c-721b-4831-8097-31443e5b34e7 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:18.474267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b189506adf4c41409cf878fe7df0bb91'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:18.474267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b189506adf4c41409cf878fe7df0bb91'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:18.477678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b189506adf4c41409cf878fe7df0bb91'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:18.493171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b189506adf4c41409cf878fe7df0bb91'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:18.509983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f04e8c-721b-4831-8097-31443e5b34e7 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:18.515324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f180896-c29c-44a1-83ff-514156af53b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:18.515865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f180896-c29c-44a1-83ff-514156af53b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:18.516153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f180896-c29c-44a1-83ff-514156af53b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:18.516425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f180896-c29c-44a1-83ff-514156af53b7 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:18.517047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f180896-c29c-44a1-83ff-514156af53b7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:18.519653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f180896-c29c-44a1-83ff-514156af53b7 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:18.640675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b189506adf4c41409cf878fe7df0bb91'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:18.647371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1009/2039] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:18 2022] DELETE /identity/v3/users/b189506adf4c41409cf878fe7df0bb91 => generated 0 bytes in 558 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:18.656631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:18.656631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:18.723292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67775a0-32fd-41ad-b952-8df1227e223d admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:18.728389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6345cf3-02eb-4bd1-b644-ab367ee359ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:18.729068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6345cf3-02eb-4bd1-b644-ab367ee359ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:18.729347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6345cf3-02eb-4bd1-b644-ab367ee359ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:18.729622 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6345cf3-02eb-4bd1-b644-ab367ee359ff admin admin] PATH_INFO: `/v3/projects/46f646fe435e4aa2954af01070537ae4/users/5247fd6c30794dc48f60a827d5837674/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:18.745559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6345cf3-02eb-4bd1-b644-ab367ee359ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=46f646fe435e4aa2954af01070537ae4, user_id=5247fd6c30794dc48f60a827d5837674, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:18.748240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6345cf3-02eb-4bd1-b644-ab367ee359ff admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:18.767015 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:18.772563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1031/2040] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:18 2022] POST /identity/v3/users => generated 288 bytes in 333 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:18.773390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f180896-c29c-44a1-83ff-514156af53b7 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:18.775037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5f180896-c29c-44a1-83ff-514156af53b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:18.787535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f180896-c29c-44a1-83ff-514156af53b7 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:18.792199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6623049-de98-4bcf-b993-42fe485cc43e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:18.816598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6623049-de98-4bcf-b993-42fe485cc43e admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:18.816598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6623049-de98-4bcf-b993-42fe485cc43e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:18.816598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6623049-de98-4bcf-b993-42fe485cc43e admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:18.816598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6623049-de98-4bcf-b993-42fe485cc43e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:18.816598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6623049-de98-4bcf-b993-42fe485cc43e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:19.106667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1032/2041] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:18 2022] POST /identity/v3/users => generated 308 bytes in 334 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:19.106796 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1010/2042] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:18 2022] PUT /identity/v3/projects/46f646fe435e4aa2954af01070537ae4/users/5247fd6c30794dc48f60a827d5837674/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:19.111026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6345cf3-02eb-4bd1-b644-ab367ee359ff admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:19.111679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6623049-de98-4bcf-b993-42fe485cc43e admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:19.114456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f6623049-de98-4bcf-b993-42fe485cc43e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:19.127225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6623049-de98-4bcf-b993-42fe485cc43e admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:19.189922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6345cf3-02eb-4bd1-b644-ab367ee359ff admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:19.195017 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548a8b04-5720-47b9-8844-79b31f6fde49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:19.195629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548a8b04-5720-47b9-8844-79b31f6fde49 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.197461 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548a8b04-5720-47b9-8844-79b31f6fde49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.197461 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548a8b04-5720-47b9-8844-79b31f6fde49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.197461 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548a8b04-5720-47b9-8844-79b31f6fde49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:19.198809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548a8b04-5720-47b9-8844-79b31f6fde49 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:19.210467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1011/2043] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:19 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:19.211737 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7112261b-1d63-4fd9-bf2b-cca8e9501a23 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.212026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7112261b-1d63-4fd9-bf2b-cca8e9501a23 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.212335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7112261b-1d63-4fd9-bf2b-cca8e9501a23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.213639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ac6b9d-d8f8-4529-bdbc-63a4b0582967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:19.214446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ac6b9d-d8f8-4529-bdbc-63a4b0582967 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.215413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ac6b9d-d8f8-4529-bdbc-63a4b0582967 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.215690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ac6b9d-d8f8-4529-bdbc-63a4b0582967 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.216403 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ac6b9d-d8f8-4529-bdbc-63a4b0582967 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:19.218487 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ac6b9d-d8f8-4529-bdbc-63a4b0582967 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:19.231210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1033/2044] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:19 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:19.232140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ac6b9d-d8f8-4529-bdbc-63a4b0582967 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:19.232603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-7112261b-1d63-4fd9-bf2b-cca8e9501a23 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:19.234502 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7112261b-1d63-4fd9-bf2b-cca8e9501a23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:19.242824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1012/2045] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:19.243687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7112261b-1d63-4fd9-bf2b-cca8e9501a23 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:19.243801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ac6b9d-d8f8-4529-bdbc-63a4b0582967 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:19.254016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7112261b-1d63-4fd9-bf2b-cca8e9501a23 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:19.257303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c49dddb-29ca-4c77-9017-42a2f06e9c82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:19.257978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c49dddb-29ca-4c77-9017-42a2f06e9c82 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.258357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c49dddb-29ca-4c77-9017-42a2f06e9c82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.258652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c49dddb-29ca-4c77-9017-42a2f06e9c82 admin admin] PATH_INFO: `/v3/projects/60fa6dcba8cf4b3db97f8de05c47be25/users/4d3fd2136f5b4e33a327f9b0a4ade8f8/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.259729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0120809c-62c2-4d7e-b5f7-fcdadadd2421 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:19.260338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0120809c-62c2-4d7e-b5f7-fcdadadd2421 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.260608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0120809c-62c2-4d7e-b5f7-fcdadadd2421 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.260865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0120809c-62c2-4d7e-b5f7-fcdadadd2421 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.261599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0120809c-62c2-4d7e-b5f7-fcdadadd2421 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:19.263189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0120809c-62c2-4d7e-b5f7-fcdadadd2421 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:19.278223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1013/2046] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:19.278223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0120809c-62c2-4d7e-b5f7-fcdadadd2421 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:19.279569 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0120809c-62c2-4d7e-b5f7-fcdadadd2421 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:19.282169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c49dddb-29ca-4c77-9017-42a2f06e9c82 admin admin] RBAC: Authorizing `identity:create_grant(project_id=60fa6dcba8cf4b3db97f8de05c47be25, user_id=4d3fd2136f5b4e33a327f9b0a4ade8f8, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:19.284925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c49dddb-29ca-4c77-9017-42a2f06e9c82 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:19.336715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0120809c-62c2-4d7e-b5f7-fcdadadd2421 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:19.340434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8dfac5-3645-4506-9e48-b82ae3740986 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:19.341031 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8dfac5-3645-4506-9e48-b82ae3740986 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.341313 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8dfac5-3645-4506-9e48-b82ae3740986 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.341586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8dfac5-3645-4506-9e48-b82ae3740986 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.350683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8dfac5-3645-4506-9e48-b82ae3740986 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:19.352114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8dfac5-3645-4506-9e48-b82ae3740986 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:19.365706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1014/2047] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:19.365706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8dfac5-3645-4506-9e48-b82ae3740986 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:19.372230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8dfac5-3645-4506-9e48-b82ae3740986 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:19.377164 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4d2a97-796e-4fe1-a209-f8a21503e1f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:19.378589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4d2a97-796e-4fe1-a209-f8a21503e1f6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.378869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4d2a97-796e-4fe1-a209-f8a21503e1f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.379133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4d2a97-796e-4fe1-a209-f8a21503e1f6 admin admin] PATH_INFO: `/v3/projects/b16c54aef0b74c6f87d551460ee691c9/users/643cb613baae43b49b0ea37eacb9bf3b/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.394227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4d2a97-796e-4fe1-a209-f8a21503e1f6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b16c54aef0b74c6f87d551460ee691c9, user_id=643cb613baae43b49b0ea37eacb9bf3b, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:19.396891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4d2a97-796e-4fe1-a209-f8a21503e1f6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:19.421558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1034/2048] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:19 2022] PUT /identity/v3/projects/60fa6dcba8cf4b3db97f8de05c47be25/users/4d3fd2136f5b4e33a327f9b0a4ade8f8/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:19.422641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c49dddb-29ca-4c77-9017-42a2f06e9c82 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:19.426238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4c49dddb-29ca-4c77-9017-42a2f06e9c82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:19.494577 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c49dddb-29ca-4c77-9017-42a2f06e9c82 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:19.499340 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:19.499977 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.500295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.500573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] PATH_INFO: `/v3/projects/1020dc139dd0423bbb5c70fd91e83da4` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.501804 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1020dc139dd0423bbb5c70fd91e83da4)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:19.504534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:19.524150 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1015/2049] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:19 2022] PUT /identity/v3/projects/b16c54aef0b74c6f87d551460ee691c9/users/643cb613baae43b49b0ea37eacb9bf3b/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:19.525955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4d2a97-796e-4fe1-a209-f8a21503e1f6 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:19.611195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4d2a97-796e-4fe1-a209-f8a21503e1f6 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:19.617680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa957e4-2dd3-4763-a178-511d38eaef56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:19.618339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa957e4-2dd3-4763-a178-511d38eaef56 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.618702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa957e4-2dd3-4763-a178-511d38eaef56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.619081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa957e4-2dd3-4763-a178-511d38eaef56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.619776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa957e4-2dd3-4763-a178-511d38eaef56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:19.621567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa957e4-2dd3-4763-a178-511d38eaef56 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:19.633373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1016/2050] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:19 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:19.634913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065dfff8-5632-4340-87db-a6a542b7721b None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.635355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065dfff8-5632-4340-87db-a6a542b7721b None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.635605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065dfff8-5632-4340-87db-a6a542b7721b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.665372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-065dfff8-5632-4340-87db-a6a542b7721b None None] MFA Rules not processed for user `643cb613baae43b49b0ea37eacb9bf3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:19.666662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-065dfff8-5632-4340-87db-a6a542b7721b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:19.701691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1020dc139dd0423bbb5c70fd91e83da4', 'request_id': 'req-70e42a4a-7a15-47e6-a497-3ef225f97921'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:19.707578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1020dc139dd0423bbb5c70fd91e83da4', 'request_id': 'req-70e42a4a-7a15-47e6-a497-3ef225f97921'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:19.734702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1017/2051] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:19 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:19.735689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065dfff8-5632-4340-87db-a6a542b7721b None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:19.747611 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065dfff8-5632-4340-87db-a6a542b7721b None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:19.791929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1020dc139dd0423bbb5c70fd91e83da4', 'request_id': 'req-70e42a4a-7a15-47e6-a497-3ef225f97921'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:19.809517 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] The token cache is being invalidate because project 1020dc139dd0423bbb5c70fd91e83da4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:19.809838 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:19.810548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:19.817435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1035/2052] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:19 2022] DELETE /identity/v3/projects/1020dc139dd0423bbb5c70fd91e83da4 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:19.817645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76258ed3-f338-4cbd-a869-2e5a9f908c04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:19.817645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76258ed3-f338-4cbd-a869-2e5a9f908c04 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.817823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:19.817898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76258ed3-f338-4cbd-a869-2e5a9f908c04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.817898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76258ed3-f338-4cbd-a869-2e5a9f908c04 admin admin] PATH_INFO: `/v3/projects/60fa6dcba8cf4b3db97f8de05c47be25/users/4d3fd2136f5b4e33a327f9b0a4ade8f8/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.818022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:19.818091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76258ed3-f338-4cbd-a869-2e5a9f908c04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=60fa6dcba8cf4b3db97f8de05c47be25, user_id=4d3fd2136f5b4e33a327f9b0a4ade8f8, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:19.819629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76258ed3-f338-4cbd-a869-2e5a9f908c04 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:19.830677 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e42a4a-7a15-47e6-a497-3ef225f97921 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:19.833046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1018/2053] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:19 2022] PUT /identity/v3/projects/60fa6dcba8cf4b3db97f8de05c47be25/users/4d3fd2136f5b4e33a327f9b0a4ade8f8/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:19.834001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7776d97-bdb8-4d53-9eba-4d79169d925c None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.834312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7776d97-bdb8-4d53-9eba-4d79169d925c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.834584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7776d97-bdb8-4d53-9eba-4d79169d925c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.860016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-b7776d97-bdb8-4d53-9eba-4d79169d925c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:19.861431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b7776d97-bdb8-4d53-9eba-4d79169d925c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:19.908476 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701874e5-d196-41b6-8427-6145a602d116 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:19.909620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701874e5-d196-41b6-8427-6145a602d116 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.909943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701874e5-d196-41b6-8427-6145a602d116 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.910453 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701874e5-d196-41b6-8427-6145a602d116 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.911091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701874e5-d196-41b6-8427-6145a602d116 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:19.913232 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701874e5-d196-41b6-8427-6145a602d116 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:19.927072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1019/2054] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:19.937020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fef51c-2b66-4daa-ae03-202d0e61ebd8 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.937307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fef51c-2b66-4daa-ae03-202d0e61ebd8 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.937578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fef51c-2b66-4daa-ae03-202d0e61ebd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.959946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-a0fef51c-2b66-4daa-ae03-202d0e61ebd8 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:19.961240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a0fef51c-2b66-4daa-ae03-202d0e61ebd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:19.974781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1020/2055] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:19.979240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fef51c-2b66-4daa-ae03-202d0e61ebd8 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:19.989262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fef51c-2b66-4daa-ae03-202d0e61ebd8 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:19.994277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c0f403-fa41-42ec-8457-9e530c86c238 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:19.994894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c0f403-fa41-42ec-8457-9e530c86c238 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:19.995167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c0f403-fa41-42ec-8457-9e530c86c238 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:19.995478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c0f403-fa41-42ec-8457-9e530c86c238 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:19.996129 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c0f403-fa41-42ec-8457-9e530c86c238 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:50:19.996457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c0f403-fa41-42ec-8457-9e530c86c238 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:19.998103 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c0f403-fa41-42ec-8457-9e530c86c238 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:20.011971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1021/2056] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:50:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:50:20.016402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c0f403-fa41-42ec-8457-9e530c86c238 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:20.028243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c0f403-fa41-42ec-8457-9e530c86c238 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:20.034876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b134d02-a4ee-47fe-91f4-35fddfed6ce0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:20.035725 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b134d02-a4ee-47fe-91f4-35fddfed6ce0 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.036012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b134d02-a4ee-47fe-91f4-35fddfed6ce0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.036282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b134d02-a4ee-47fe-91f4-35fddfed6ce0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.036917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b134d02-a4ee-47fe-91f4-35fddfed6ce0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:20.038881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b134d02-a4ee-47fe-91f4-35fddfed6ce0 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:20.062220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1036/2057] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:19 2022] POST /identity/v3/projects => generated 377 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:20.067388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701874e5-d196-41b6-8427-6145a602d116 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:20.134369 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701874e5-d196-41b6-8427-6145a602d116 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:20.139103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcee148-283b-450b-b790-1c0a208f4535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:20.139720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcee148-283b-450b-b790-1c0a208f4535 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.140037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcee148-283b-450b-b790-1c0a208f4535 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.140287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcee148-283b-450b-b790-1c0a208f4535 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.140910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcee148-283b-450b-b790-1c0a208f4535 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:20.142931 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcee148-283b-450b-b790-1c0a208f4535 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:20.153487 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1022/2058] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:20 2022] POST /identity/v3/projects => generated 367 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:20.162691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b134d02-a4ee-47fe-91f4-35fddfed6ce0 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:20.242666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b134d02-a4ee-47fe-91f4-35fddfed6ce0 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:20.247534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45256a7b-740d-4125-b273-0ef3e19b589e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:20.248144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45256a7b-740d-4125-b273-0ef3e19b589e admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.248421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45256a7b-740d-4125-b273-0ef3e19b589e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.248694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45256a7b-740d-4125-b273-0ef3e19b589e admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.249418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45256a7b-740d-4125-b273-0ef3e19b589e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:20.251403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45256a7b-740d-4125-b273-0ef3e19b589e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:20.279205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1037/2059] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:20 2022] POST /identity/v3/users => generated 289 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:20.280768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcee148-283b-450b-b790-1c0a208f4535 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:20.291286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcee148-283b-450b-b790-1c0a208f4535 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:20.302053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5365f95-e7a4-4afb-9030-53a600dc7c09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:20.302718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5365f95-e7a4-4afb-9030-53a600dc7c09 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.303013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5365f95-e7a4-4afb-9030-53a600dc7c09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.303336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5365f95-e7a4-4afb-9030-53a600dc7c09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.304141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5365f95-e7a4-4afb-9030-53a600dc7c09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:20.305960 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5365f95-e7a4-4afb-9030-53a600dc7c09 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:20.318084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1038/2060] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:20 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:50:20.322193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5365f95-e7a4-4afb-9030-53a600dc7c09 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:20.332413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5365f95-e7a4-4afb-9030-53a600dc7c09 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:20.338754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a33b232-653a-4637-8e23-116f84bcf2d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:20.339372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a33b232-653a-4637-8e23-116f84bcf2d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.339651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a33b232-653a-4637-8e23-116f84bcf2d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.339925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a33b232-653a-4637-8e23-116f84bcf2d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.340585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a33b232-653a-4637-8e23-116f84bcf2d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:20.342228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a33b232-653a-4637-8e23-116f84bcf2d1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:20.352885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1039/2061] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:50:20.357052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a33b232-653a-4637-8e23-116f84bcf2d1 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:20.368101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a33b232-653a-4637-8e23-116f84bcf2d1 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:20.370197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1023/2062] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:20 2022] POST /identity/v3/users => generated 284 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:50:20.377336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0af424-c747-4c0f-902a-6cab7ad8581f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:20.377336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0af424-c747-4c0f-902a-6cab7ad8581f admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.377336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0af424-c747-4c0f-902a-6cab7ad8581f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.377336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0af424-c747-4c0f-902a-6cab7ad8581f admin admin] PATH_INFO: `/v3/projects/dc8001ce88eb43e19c1363d824896246/users/c1d91595ccb74e6b9996f9e00540a127/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.379565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45256a7b-740d-4125-b273-0ef3e19b589e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:20.388901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45256a7b-740d-4125-b273-0ef3e19b589e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:20.394791 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063883ce-97b9-4ed6-8e68-c8c142f838ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:20.395660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063883ce-97b9-4ed6-8e68-c8c142f838ae admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.396025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063883ce-97b9-4ed6-8e68-c8c142f838ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.396301 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063883ce-97b9-4ed6-8e68-c8c142f838ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.397148 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063883ce-97b9-4ed6-8e68-c8c142f838ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:20.398047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0af424-c747-4c0f-902a-6cab7ad8581f admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc8001ce88eb43e19c1363d824896246, user_id=c1d91595ccb74e6b9996f9e00540a127, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:20.398639 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063883ce-97b9-4ed6-8e68-c8c142f838ae admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:20.401326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0af424-c747-4c0f-902a-6cab7ad8581f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:20.414388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1024/2063] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:20 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:20.415407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063883ce-97b9-4ed6-8e68-c8c142f838ae admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:20.427654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063883ce-97b9-4ed6-8e68-c8c142f838ae admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:20.432835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb86cee0-76ed-4eac-b06b-cbcb9e3fda14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:20.433629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb86cee0-76ed-4eac-b06b-cbcb9e3fda14 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.433916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb86cee0-76ed-4eac-b06b-cbcb9e3fda14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.434380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb86cee0-76ed-4eac-b06b-cbcb9e3fda14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.435109 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb86cee0-76ed-4eac-b06b-cbcb9e3fda14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:20.436768 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb86cee0-76ed-4eac-b06b-cbcb9e3fda14 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:20.449275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1025/2064] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:20 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:20.458689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb86cee0-76ed-4eac-b06b-cbcb9e3fda14 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:20.469004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb86cee0-76ed-4eac-b06b-cbcb9e3fda14 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:20.474632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5f8989-8c56-407e-802d-e234668dcb28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:20.480983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5f8989-8c56-407e-802d-e234668dcb28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.480983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5f8989-8c56-407e-802d-e234668dcb28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.480983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5f8989-8c56-407e-802d-e234668dcb28 admin admin] PATH_INFO: `/v3/projects/6552bf143d28461c98362adc695d06a3/users/40c5fca261e24b49adca1b04fdcba5c6/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.492430 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5f8989-8c56-407e-802d-e234668dcb28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6552bf143d28461c98362adc695d06a3, user_id=40c5fca261e24b49adca1b04fdcba5c6, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:20.495482 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5f8989-8c56-407e-802d-e234668dcb28 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:20.543201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1040/2065] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:20 2022] PUT /identity/v3/projects/dc8001ce88eb43e19c1363d824896246/users/c1d91595ccb74e6b9996f9e00540a127/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:20.555824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a6fc8c-3f15-43dd-aead-17e7fdb0e551 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.556123 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a6fc8c-3f15-43dd-aead-17e7fdb0e551 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.556391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a6fc8c-3f15-43dd-aead-17e7fdb0e551 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.595459 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-45a6fc8c-3f15-43dd-aead-17e7fdb0e551 None None] MFA Rules not processed for user `c1d91595ccb74e6b9996f9e00540a127`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:20.597556 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-45a6fc8c-3f15-43dd-aead-17e7fdb0e551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:20.610313 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1026/2066] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:20 2022] PUT /identity/v3/projects/6552bf143d28461c98362adc695d06a3/users/40c5fca261e24b49adca1b04fdcba5c6/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:20.623871 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e9b671-cccf-480c-8667-497fdffa3f68 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.624268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e9b671-cccf-480c-8667-497fdffa3f68 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.624827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e9b671-cccf-480c-8667-497fdffa3f68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.681205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1041/2067] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:20.682145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a6fc8c-3f15-43dd-aead-17e7fdb0e551 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:20.685069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-69e9b671-cccf-480c-8667-497fdffa3f68 None None] MFA Rules not processed for user `40c5fca261e24b49adca1b04fdcba5c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:20.686786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-69e9b671-cccf-480c-8667-497fdffa3f68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:20.774032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a6fc8c-3f15-43dd-aead-17e7fdb0e551 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:20.821473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1027/2068] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:20 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:20.837474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e9b671-cccf-480c-8667-497fdffa3f68 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:20.840534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-69e9b671-cccf-480c-8667-497fdffa3f68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:20.875973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79464b23-c2b5-4495-bc94-b89a44fb4b87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:20.876652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79464b23-c2b5-4495-bc94-b89a44fb4b87 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.876941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79464b23-c2b5-4495-bc94-b89a44fb4b87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.877215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79464b23-c2b5-4495-bc94-b89a44fb4b87 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.878566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79464b23-c2b5-4495-bc94-b89a44fb4b87 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:20.880696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79464b23-c2b5-4495-bc94-b89a44fb4b87 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:20.929024 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e9b671-cccf-480c-8667-497fdffa3f68 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:20.933003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbf5e20-e944-480d-87c1-a96cb60239f5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f3b3c7ebc08496e970f8f25fd6395e4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:20.933695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbf5e20-e944-480d-87c1-a96cb60239f5 service glance] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:20.934043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbf5e20-e944-480d-87c1-a96cb60239f5 service glance] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:20.934444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbf5e20-e944-480d-87c1-a96cb60239f5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:20.946897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbf5e20-e944-480d-87c1-a96cb60239f5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:20.948511 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbf5e20-e944-480d-87c1-a96cb60239f5 service glance] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:20.958498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1028/2069] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:20 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:21.080313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1042/2070] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:20 2022] POST /identity/v3/projects => generated 377 bytes in 395 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:21.083365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbf5e20-e944-480d-87c1-a96cb60239f5 service glance] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:21.100493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbf5e20-e944-480d-87c1-a96cb60239f5 service glance] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:21.197289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:21.222123 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac18276-7426-4233-85cd-2fd7905eb9a9 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:21.226620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac18276-7426-4233-85cd-2fd7905eb9a9 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:21.227031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac18276-7426-4233-85cd-2fd7905eb9a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:21.246736 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a529f0a-8d84-4ef8-b070-8bef7f7fcb10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:21.249193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a529f0a-8d84-4ef8-b070-8bef7f7fcb10 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:21.249193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a529f0a-8d84-4ef8-b070-8bef7f7fcb10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:21.249193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a529f0a-8d84-4ef8-b070-8bef7f7fcb10 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:21.249193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a529f0a-8d84-4ef8-b070-8bef7f7fcb10 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:21.253708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a529f0a-8d84-4ef8-b070-8bef7f7fcb10 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:21.279727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-dac18276-7426-4233-85cd-2fd7905eb9a9 None None] MFA Rules not processed for user `5247fd6c30794dc48f60a827d5837674`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:21.281640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-dac18276-7426-4233-85cd-2fd7905eb9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:21.367224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1029/2071] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:21 2022] POST /identity/v3/users => generated 295 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:21.371068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a529f0a-8d84-4ef8-b070-8bef7f7fcb10 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:21.395650 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a529f0a-8d84-4ef8-b070-8bef7f7fcb10 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:21.402640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1043/2072] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:21.405770 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca192d8-301b-480e-9914-973945d74485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:21.411818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca192d8-301b-480e-9914-973945d74485 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:21.411818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca192d8-301b-480e-9914-973945d74485 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:21.411818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca192d8-301b-480e-9914-973945d74485 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:21.411818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca192d8-301b-480e-9914-973945d74485 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:21.411818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca192d8-301b-480e-9914-973945d74485 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:21.430372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1030/2073] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:21 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:21.435297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac18276-7426-4233-85cd-2fd7905eb9a9 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:21.435951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca192d8-301b-480e-9914-973945d74485 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:21.437957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0ca192d8-301b-480e-9914-973945d74485 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:21.453549 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac18276-7426-4233-85cd-2fd7905eb9a9 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:21.462262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94660f85-8302-433c-bbc0-ed705d76694d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:21.463118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94660f85-8302-433c-bbc0-ed705d76694d admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:21.463484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94660f85-8302-433c-bbc0-ed705d76694d admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:21.463866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94660f85-8302-433c-bbc0-ed705d76694d admin admin] PATH_INFO: `/v3/projects/547074bf4b2c48258e00d5bf8c174f1e/users/dae7705cc7a34e828f4d4ce5f8583d5e/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:21.478326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:21.486476 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94660f85-8302-433c-bbc0-ed705d76694d admin admin] RBAC: Authorizing `identity:create_grant(project_id=547074bf4b2c48258e00d5bf8c174f1e, user_id=dae7705cc7a34e828f4d4ce5f8583d5e, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:21.490968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94660f85-8302-433c-bbc0-ed705d76694d admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:21.539115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca192d8-301b-480e-9914-973945d74485 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:21.545490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367c2d06-5492-4985-a6b7-93c0136389c1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:21.546457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367c2d06-5492-4985-a6b7-93c0136389c1 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:21.546920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367c2d06-5492-4985-a6b7-93c0136389c1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:21.547341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367c2d06-5492-4985-a6b7-93c0136389c1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:21.568089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367c2d06-5492-4985-a6b7-93c0136389c1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:21.569845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367c2d06-5492-4985-a6b7-93c0136389c1 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:21.580027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1044/2074] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:21 2022] PUT /identity/v3/projects/547074bf4b2c48258e00d5bf8c174f1e/users/dae7705cc7a34e828f4d4ce5f8583d5e/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:21.582699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1031/2075] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:21 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:21.587561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94660f85-8302-433c-bbc0-ed705d76694d admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:21.685451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94660f85-8302-433c-bbc0-ed705d76694d admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:21.691261 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b67a8b-a74d-4403-97e9-180005633924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:21.691857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b67a8b-a74d-4403-97e9-180005633924 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:21.692196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b67a8b-a74d-4403-97e9-180005633924 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:21.692475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b67a8b-a74d-4403-97e9-180005633924 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:21.693134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b67a8b-a74d-4403-97e9-180005633924 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:21.694875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b67a8b-a74d-4403-97e9-180005633924 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:21.709861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1045/2076] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:21 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:21.711645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367c2d06-5492-4985-a6b7-93c0136389c1 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:21.727636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367c2d06-5492-4985-a6b7-93c0136389c1 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:21.733491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e023e8-0647-4cd3-bba0-c217847a69e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:21.734362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e023e8-0647-4cd3-bba0-c217847a69e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:21.734649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e023e8-0647-4cd3-bba0-c217847a69e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:21.734916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e023e8-0647-4cd3-bba0-c217847a69e2 admin admin] PATH_INFO: `/v3/projects/547074bf4b2c48258e00d5bf8c174f1e/users/dae7705cc7a34e828f4d4ce5f8583d5e/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:21.737681 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e023e8-0647-4cd3-bba0-c217847a69e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=547074bf4b2c48258e00d5bf8c174f1e, user_id=dae7705cc7a34e828f4d4ce5f8583d5e, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:21.746925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e023e8-0647-4cd3-bba0-c217847a69e2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:21.760100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1032/2077] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:21 2022] PUT /identity/v3/projects/547074bf4b2c48258e00d5bf8c174f1e/users/dae7705cc7a34e828f4d4ce5f8583d5e/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:21.770671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a24f60-80f1-4c48-ac11-37c5c623e551 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:21.770963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a24f60-80f1-4c48-ac11-37c5c623e551 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:21.771246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a24f60-80f1-4c48-ac11-37c5c623e551 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:21.803252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-f1a24f60-80f1-4c48-ac11-37c5c623e551 None None] MFA Rules not processed for user `dae7705cc7a34e828f4d4ce5f8583d5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:21.804582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f1a24f60-80f1-4c48-ac11-37c5c623e551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:21.891947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1046/2078] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:21 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:21.965851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a49c5a6-3ac6-4cf0-8028-896b9d4199cf None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:21.966135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a49c5a6-3ac6-4cf0-8028-896b9d4199cf None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:21.966440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a49c5a6-3ac6-4cf0-8028-896b9d4199cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:22.001399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-0a49c5a6-3ac6-4cf0-8028-896b9d4199cf None None] MFA Rules not processed for user `4d3fd2136f5b4e33a327f9b0a4ade8f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:22.003449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0a49c5a6-3ac6-4cf0-8028-896b9d4199cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:22.091193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1033/2079] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:21 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:22.176835 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a24f60-80f1-4c48-ac11-37c5c623e551 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:22.178907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f1a24f60-80f1-4c48-ac11-37c5c623e551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:22.242290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a24f60-80f1-4c48-ac11-37c5c623e551 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:22.246532 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802d267-1a64-4e92-90b0-9f5454e1c8f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:22.246909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802d267-1a64-4e92-90b0-9f5454e1c8f3 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:22.247190 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802d267-1a64-4e92-90b0-9f5454e1c8f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:22.247467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802d267-1a64-4e92-90b0-9f5454e1c8f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:22.249239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0802d267-1a64-4e92-90b0-9f5454e1c8f3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:22.331914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0802d267-1a64-4e92-90b0-9f5454e1c8f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:22.334101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0802d267-1a64-4e92-90b0-9f5454e1c8f3 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:22.346173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1047/2080] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:22 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:23.388487 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:23.764513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a49c5a6-3ac6-4cf0-8028-896b9d4199cf None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:23.766072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0a49c5a6-3ac6-4cf0-8028-896b9d4199cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:23.834332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a49c5a6-3ac6-4cf0-8028-896b9d4199cf None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:23.838964 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0aad409-5657-487b-9ca2-df4919fb7d07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:23.839599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0aad409-5657-487b-9ca2-df4919fb7d07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:23.839880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0aad409-5657-487b-9ca2-df4919fb7d07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:23.840154 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0aad409-5657-487b-9ca2-df4919fb7d07 admin admin] PATH_INFO: `/v3/users/abad1328adbb484c9478fa8e2c1817af` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:23.841353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0aad409-5657-487b-9ca2-df4919fb7d07 admin admin] RBAC: Authorizing `identity:delete_user(user_id=abad1328adbb484c9478fa8e2c1817af)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:23.843351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0aad409-5657-487b-9ca2-df4919fb7d07 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:23.951934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:DELETE=1,keystone:SELECT=91,keystone:INSERT=8 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:24.028215 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a0aad409-5657-487b-9ca2-df4919fb7d07 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'abad1328adbb484c9478fa8e2c1817af'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:24.028811 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a0aad409-5657-487b-9ca2-df4919fb7d07 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'abad1328adbb484c9478fa8e2c1817af'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:24.038210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a0aad409-5657-487b-9ca2-df4919fb7d07 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'abad1328adbb484c9478fa8e2c1817af'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:24.052687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a0aad409-5657-487b-9ca2-df4919fb7d07 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'abad1328adbb484c9478fa8e2c1817af'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:24.111325 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a0aad409-5657-487b-9ca2-df4919fb7d07 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'abad1328adbb484c9478fa8e2c1817af'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:24.121485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1034/2081] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:23 2022] DELETE /identity/v3/users/abad1328adbb484c9478fa8e2c1817af => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:24.398857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802d267-1a64-4e92-90b0-9f5454e1c8f3 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:24.413445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802d267-1a64-4e92-90b0-9f5454e1c8f3 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:24.484834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:24.488225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:24.497633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:24.497633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] PATH_INFO: `/v3/projects/806e734529f54d569ef68b63587f704f` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:24.497633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] RBAC: Authorizing `identity:delete_project(project_id=806e734529f54d569ef68b63587f704f)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:24.497633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:24.558124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1237c0-6d9f-4748-a672-1e3f3c77bde2 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:24.558793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1237c0-6d9f-4748-a672-1e3f3c77bde2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:24.558793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1237c0-6d9f-4748-a672-1e3f3c77bde2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:24.580514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-bc1237c0-6d9f-4748-a672-1e3f3c77bde2 None None] MFA Rules not processed for user `5247fd6c30794dc48f60a827d5837674`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:24.582126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-bc1237c0-6d9f-4748-a672-1e3f3c77bde2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:24.638133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '806e734529f54d569ef68b63587f704f', 'request_id': 'req-6a1e83c4-642a-4a79-8524-83c6a9ed0970'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:24.641954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1035/2082] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:24.644539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '806e734529f54d569ef68b63587f704f', 'request_id': 'req-6a1e83c4-642a-4a79-8524-83c6a9ed0970'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:24.743534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '806e734529f54d569ef68b63587f704f', 'request_id': 'req-6a1e83c4-642a-4a79-8524-83c6a9ed0970'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:24.758365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] The token cache is being invalidate because project 806e734529f54d569ef68b63587f704f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:24.758654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:24.759326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:24.760921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1048/2083] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:24 2022] DELETE /identity/v3/projects/806e734529f54d569ef68b63587f704f => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:24.765160 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1237c0-6d9f-4748-a672-1e3f3c77bde2 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:24.766627 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-bc1237c0-6d9f-4748-a672-1e3f3c77bde2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:24.782063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1237c0-6d9f-4748-a672-1e3f3c77bde2 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:24.844957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:24.845600 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:24.845877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:24.846260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] PATH_INFO: `/v3/users/4d3fd2136f5b4e33a327f9b0a4ade8f8` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:24.847668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4d3fd2136f5b4e33a327f9b0a4ade8f8)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:24.849656 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:25.084580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4d3fd2136f5b4e33a327f9b0a4ade8f8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:25.084875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4d3fd2136f5b4e33a327f9b0a4ade8f8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:25.094505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4d3fd2136f5b4e33a327f9b0a4ade8f8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:25.110389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4d3fd2136f5b4e33a327f9b0a4ade8f8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:25.196576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4d3fd2136f5b4e33a327f9b0a4ade8f8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:25.203371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1036/2084] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:24 2022] DELETE /identity/v3/users/4d3fd2136f5b4e33a327f9b0a4ade8f8 => generated 0 bytes in 439 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:25.554254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:25.555749 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:25.578507 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:25.589572 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1e83c4-642a-4a79-8524-83c6a9ed0970 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:25.670740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f145d904-0706-46ab-a94b-d835c03fe61a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:25.671482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f145d904-0706-46ab-a94b-d835c03fe61a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:25.671785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f145d904-0706-46ab-a94b-d835c03fe61a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:25.672064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f145d904-0706-46ab-a94b-d835c03fe61a admin admin] PATH_INFO: `/v3/projects/60fa6dcba8cf4b3db97f8de05c47be25` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:25.673543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f145d904-0706-46ab-a94b-d835c03fe61a admin admin] RBAC: Authorizing `identity:delete_project(project_id=60fa6dcba8cf4b3db97f8de05c47be25)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:25.676172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f145d904-0706-46ab-a94b-d835c03fe61a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:25.749342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:25.750788 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:25.813697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cba03d-31e7-40a5-a8a3-1831283b0214 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:25.817606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ea54d6-2d04-4c0d-81da-6ddf70ac82a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:25.818263 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ea54d6-2d04-4c0d-81da-6ddf70ac82a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:25.818544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ea54d6-2d04-4c0d-81da-6ddf70ac82a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:25.818817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ea54d6-2d04-4c0d-81da-6ddf70ac82a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:25.820604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c4ea54d6-2d04-4c0d-81da-6ddf70ac82a7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:25.886987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ea54d6-2d04-4c0d-81da-6ddf70ac82a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:25.888520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ea54d6-2d04-4c0d-81da-6ddf70ac82a7 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:25.894679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f145d904-0706-46ab-a94b-d835c03fe61a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '60fa6dcba8cf4b3db97f8de05c47be25', 'request_id': 'req-f145d904-0706-46ab-a94b-d835c03fe61a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:25.898397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1037/2085] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:25 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:50:25.902842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f145d904-0706-46ab-a94b-d835c03fe61a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '60fa6dcba8cf4b3db97f8de05c47be25', 'request_id': 'req-f145d904-0706-46ab-a94b-d835c03fe61a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:26.000433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f145d904-0706-46ab-a94b-d835c03fe61a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '60fa6dcba8cf4b3db97f8de05c47be25', 'request_id': 'req-f145d904-0706-46ab-a94b-d835c03fe61a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:26.014385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f145d904-0706-46ab-a94b-d835c03fe61a admin admin] The token cache is being invalidate because project 60fa6dcba8cf4b3db97f8de05c47be25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:26.014602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f145d904-0706-46ab-a94b-d835c03fe61a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:26.015259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f145d904-0706-46ab-a94b-d835c03fe61a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:26.016887 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1049/2086] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:25 2022] DELETE /identity/v3/projects/60fa6dcba8cf4b3db97f8de05c47be25 => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:26.022689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ef7b36-072d-4713-9f77-8123e406f32f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:26.022976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ef7b36-072d-4713-9f77-8123e406f32f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:26.023247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ef7b36-072d-4713-9f77-8123e406f32f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:26.043688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-a4ef7b36-072d-4713-9f77-8123e406f32f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:26.045041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a4ef7b36-072d-4713-9f77-8123e406f32f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:26.119687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1038/2087] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:26.129525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83277c9f-93db-4716-8340-8b93e3460815 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:26.129831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83277c9f-93db-4716-8340-8b93e3460815 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:26.130065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83277c9f-93db-4716-8340-8b93e3460815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:26.139735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:26.152760 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-83277c9f-93db-4716-8340-8b93e3460815 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:26.154071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-83277c9f-93db-4716-8340-8b93e3460815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:26.162689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1050/2088] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:26.167283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ef7b36-072d-4713-9f77-8123e406f32f None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:26.180849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ef7b36-072d-4713-9f77-8123e406f32f None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:26.186011 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ad297c-c91b-4462-8ada-11f4c63834ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:26.186657 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ad297c-c91b-4462-8ada-11f4c63834ad admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:26.186933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ad297c-c91b-4462-8ada-11f4c63834ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:26.187202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ad297c-c91b-4462-8ada-11f4c63834ad admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:26.187877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ad297c-c91b-4462-8ada-11f4c63834ad admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:50:26.188175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ad297c-c91b-4462-8ada-11f4c63834ad admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:26.190552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ad297c-c91b-4462-8ada-11f4c63834ad admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:26.206187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1039/2089] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:50:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:50:26.210836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83277c9f-93db-4716-8340-8b93e3460815 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:26.222330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83277c9f-93db-4716-8340-8b93e3460815 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:26.229822 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ede5718-5f72-403f-8c01-d6f81d100cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:26.230486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ede5718-5f72-403f-8c01-d6f81d100cf8 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:26.230759 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ede5718-5f72-403f-8c01-d6f81d100cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:26.231026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ede5718-5f72-403f-8c01-d6f81d100cf8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:26.231663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ede5718-5f72-403f-8c01-d6f81d100cf8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:26.233596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ede5718-5f72-403f-8c01-d6f81d100cf8 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:26.385794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1051/2090] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:26 2022] POST /identity/v3/projects => generated 407 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:26.390681 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ad297c-c91b-4462-8ada-11f4c63834ad admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:26.403892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ad297c-c91b-4462-8ada-11f4c63834ad admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:26.465787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92bf01a-767d-4c07-b450-7e932be5614b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:26.466501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92bf01a-767d-4c07-b450-7e932be5614b admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:26.466784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92bf01a-767d-4c07-b450-7e932be5614b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:26.467061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92bf01a-767d-4c07-b450-7e932be5614b admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:26.467741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92bf01a-767d-4c07-b450-7e932be5614b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:26.469694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92bf01a-767d-4c07-b450-7e932be5614b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:26.620353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1040/2091] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:26 2022] POST /identity/v3/users => generated 304 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:50:26.626319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ede5718-5f72-403f-8c01-d6f81d100cf8 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:26.640650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ede5718-5f72-403f-8c01-d6f81d100cf8 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:26.650084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d235e9-91a2-4b8e-8eb8-7cb3fecdc8fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:26.651913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d235e9-91a2-4b8e-8eb8-7cb3fecdc8fb admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:26.651913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d235e9-91a2-4b8e-8eb8-7cb3fecdc8fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:26.651913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d235e9-91a2-4b8e-8eb8-7cb3fecdc8fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:26.652897 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d235e9-91a2-4b8e-8eb8-7cb3fecdc8fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:26.654787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d235e9-91a2-4b8e-8eb8-7cb3fecdc8fb admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:26.666573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1052/2092] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:26 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:50:26.670940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92bf01a-767d-4c07-b450-7e932be5614b admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:26.681689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92bf01a-767d-4c07-b450-7e932be5614b admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:26.691969 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50d0cb6-58e7-455d-9e5c-ff4c14dfbb87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:26.692600 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50d0cb6-58e7-455d-9e5c-ff4c14dfbb87 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:26.692880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50d0cb6-58e7-455d-9e5c-ff4c14dfbb87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:26.693151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50d0cb6-58e7-455d-9e5c-ff4c14dfbb87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:26.693971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50d0cb6-58e7-455d-9e5c-ff4c14dfbb87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:26.695690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50d0cb6-58e7-455d-9e5c-ff4c14dfbb87 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:26.711105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1041/2093] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:26 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:50:26.712289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d235e9-91a2-4b8e-8eb8-7cb3fecdc8fb admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:26.722725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d235e9-91a2-4b8e-8eb8-7cb3fecdc8fb admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:26.728563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b36c41-245c-44aa-8802-260cabe5774c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:26.732289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b36c41-245c-44aa-8802-260cabe5774c admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:26.732289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b36c41-245c-44aa-8802-260cabe5774c admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:26.732289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b36c41-245c-44aa-8802-260cabe5774c admin admin] PATH_INFO: `/v3/projects/520b762c22a44f1485ce89b426de7a57/users/d53a359ab87b494dae7817665634551c/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:26.753189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b36c41-245c-44aa-8802-260cabe5774c admin admin] RBAC: Authorizing `identity:create_grant(project_id=520b762c22a44f1485ce89b426de7a57, user_id=d53a359ab87b494dae7817665634551c, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:26.756413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b36c41-245c-44aa-8802-260cabe5774c admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:26.875093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1053/2094] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:26 2022] PUT /identity/v3/projects/520b762c22a44f1485ce89b426de7a57/users/d53a359ab87b494dae7817665634551c/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:26.886497 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95f06ee-29a5-4c25-a6b7-0e5196289761 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:26.886785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95f06ee-29a5-4c25-a6b7-0e5196289761 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:26.887060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95f06ee-29a5-4c25-a6b7-0e5196289761 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:26.919558 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-e95f06ee-29a5-4c25-a6b7-0e5196289761 None None] MFA Rules not processed for user `d53a359ab87b494dae7817665634551c`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:26.920828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-e95f06ee-29a5-4c25-a6b7-0e5196289761 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:26.995424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1042/2095] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:26 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:27.000339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b36c41-245c-44aa-8802-260cabe5774c admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:27.003544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:27.068955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b36c41-245c-44aa-8802-260cabe5774c admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:27.079106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d79aea3-1904-41e2-b7ba-ac52ad2c3274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:27.079773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d79aea3-1904-41e2-b7ba-ac52ad2c3274 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:27.080058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d79aea3-1904-41e2-b7ba-ac52ad2c3274 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:27.080331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d79aea3-1904-41e2-b7ba-ac52ad2c3274 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:27.081034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d79aea3-1904-41e2-b7ba-ac52ad2c3274 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:27.082983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d79aea3-1904-41e2-b7ba-ac52ad2c3274 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:27.224244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1054/2096] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:26 2022] POST /identity/v3/projects => generated 409 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:27.228180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95f06ee-29a5-4c25-a6b7-0e5196289761 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:27.245608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95f06ee-29a5-4c25-a6b7-0e5196289761 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:27.319458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90b8d16-6f36-48c4-ad57-3bc6e9ce5e06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:27.320120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90b8d16-6f36-48c4-ad57-3bc6e9ce5e06 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:27.320399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90b8d16-6f36-48c4-ad57-3bc6e9ce5e06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:27.320673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90b8d16-6f36-48c4-ad57-3bc6e9ce5e06 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:27.321524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90b8d16-6f36-48c4-ad57-3bc6e9ce5e06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:27.323517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90b8d16-6f36-48c4-ad57-3bc6e9ce5e06 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:27.432626 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1043/2097] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:27 2022] POST /identity/v3/users => generated 311 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:27.437177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d79aea3-1904-41e2-b7ba-ac52ad2c3274 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:27.449008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d79aea3-1904-41e2-b7ba-ac52ad2c3274 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:27.455008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a122532-39ff-40e8-a2cc-e0833fedb875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:27.455680 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a122532-39ff-40e8-a2cc-e0833fedb875 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:27.456049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a122532-39ff-40e8-a2cc-e0833fedb875 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:27.456395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a122532-39ff-40e8-a2cc-e0833fedb875 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:27.457034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a122532-39ff-40e8-a2cc-e0833fedb875 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:27.458717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a122532-39ff-40e8-a2cc-e0833fedb875 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:27.477419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1055/2098] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:27 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:50:27.478839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90b8d16-6f36-48c4-ad57-3bc6e9ce5e06 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:27.491074 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90b8d16-6f36-48c4-ad57-3bc6e9ce5e06 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:27.495837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa07916-c445-4c92-ae4d-154af3a6ae55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:27.496501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa07916-c445-4c92-ae4d-154af3a6ae55 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:27.496779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa07916-c445-4c92-ae4d-154af3a6ae55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:27.497052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa07916-c445-4c92-ae4d-154af3a6ae55 admin admin] PATH_INFO: `/v3/projects/7f19139273854430a865796391971664/users/4a4ea601eae144f08ab44953935279b3/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:27.513179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa07916-c445-4c92-ae4d-154af3a6ae55 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f19139273854430a865796391971664, user_id=4a4ea601eae144f08ab44953935279b3, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:27.516722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa07916-c445-4c92-ae4d-154af3a6ae55 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:27.645976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1044/2099] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:27 2022] PUT /identity/v3/projects/7f19139273854430a865796391971664/users/4a4ea601eae144f08ab44953935279b3/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:27.653840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a122532-39ff-40e8-a2cc-e0833fedb875 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:27.666965 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a122532-39ff-40e8-a2cc-e0833fedb875 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:27.746009 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa07916-c445-4c92-ae4d-154af3a6ae55 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:27.746009 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-2aa07916-c445-4c92-ae4d-154af3a6ae55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:27.763917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0155cf6f-fe67-4a55-99f9-075a9d5f87f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:27.764637 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0155cf6f-fe67-4a55-99f9-075a9d5f87f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:27.764983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0155cf6f-fe67-4a55-99f9-075a9d5f87f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:27.765347 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0155cf6f-fe67-4a55-99f9-075a9d5f87f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:27.766067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0155cf6f-fe67-4a55-99f9-075a9d5f87f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:27.768852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0155cf6f-fe67-4a55-99f9-075a9d5f87f0 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:27.778950 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1056/2100] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:27 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:27.785849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0155cf6f-fe67-4a55-99f9-075a9d5f87f0 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:27.798194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0155cf6f-fe67-4a55-99f9-075a9d5f87f0 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:27.802975 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306e3ea1-891e-4812-a664-826027058a5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:27.803647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306e3ea1-891e-4812-a664-826027058a5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:27.803924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306e3ea1-891e-4812-a664-826027058a5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:27.804203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306e3ea1-891e-4812-a664-826027058a5a admin admin] PATH_INFO: `/v3/projects/7f19139273854430a865796391971664/users/4a4ea601eae144f08ab44953935279b3/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:27.806336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306e3ea1-891e-4812-a664-826027058a5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f19139273854430a865796391971664, user_id=4a4ea601eae144f08ab44953935279b3, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:27.808994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306e3ea1-891e-4812-a664-826027058a5a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:27.812947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa07916-c445-4c92-ae4d-154af3a6ae55 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:27.817628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b43e1e-0c8a-410c-aac5-757d9c422a72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:27.818484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b43e1e-0c8a-410c-aac5-757d9c422a72 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:27.819465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b43e1e-0c8a-410c-aac5-757d9c422a72 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:27.819465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b43e1e-0c8a-410c-aac5-757d9c422a72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:27.821087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a6b43e1e-0c8a-410c-aac5-757d9c422a72 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:27.822298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1057/2101] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:27 2022] PUT /identity/v3/projects/7f19139273854430a865796391971664/users/4a4ea601eae144f08ab44953935279b3/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:27.834421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27005673-aaaf-48c3-831e-25e50feba22b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:27.834710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27005673-aaaf-48c3-831e-25e50feba22b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:27.834989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27005673-aaaf-48c3-831e-25e50feba22b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:27.854417 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:27.877888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-27005673-aaaf-48c3-831e-25e50feba22b None None] MFA Rules not processed for user `4a4ea601eae144f08ab44953935279b3`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:27.879249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-27005673-aaaf-48c3-831e-25e50feba22b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:27.904084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b43e1e-0c8a-410c-aac5-757d9c422a72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:27.905555 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b43e1e-0c8a-410c-aac5-757d9c422a72 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:27.917921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1045/2102] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:27 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:50:27.981869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1058/2103] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:27 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:27.993184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b43e1e-0c8a-410c-aac5-757d9c422a72 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:27.994651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a6b43e1e-0c8a-410c-aac5-757d9c422a72 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:28.008184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b43e1e-0c8a-410c-aac5-757d9c422a72 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:28.060455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ffec1e-6b34-4737-b62b-f7152294ad0b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:28.061093 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ffec1e-6b34-4737-b62b-f7152294ad0b service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:28.061387 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ffec1e-6b34-4737-b62b-f7152294ad0b service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:28.061659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ffec1e-6b34-4737-b62b-f7152294ad0b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:28.072313 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ffec1e-6b34-4737-b62b-f7152294ad0b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:28.073769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ffec1e-6b34-4737-b62b-f7152294ad0b service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:28.084916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1046/2104] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:27 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:28.151342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:28.534460 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27005673-aaaf-48c3-831e-25e50feba22b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:28.544228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27005673-aaaf-48c3-831e-25e50feba22b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:28.547989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bba94d6-0be4-4518-9291-0bded7fde796 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:28.548634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bba94d6-0be4-4518-9291-0bded7fde796 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:28.548915 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bba94d6-0be4-4518-9291-0bded7fde796 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:28.549933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bba94d6-0be4-4518-9291-0bded7fde796 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:28.617641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bba94d6-0be4-4518-9291-0bded7fde796 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:28.619450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bba94d6-0be4-4518-9291-0bded7fde796 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:28.630605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1059/2105] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:28 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:28.905730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:29.385005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:30.014792 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:30.288520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:30.677097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:31.043547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:31.510074 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:32.179486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:32.786061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:32.874828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:38.426275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ffec1e-6b34-4737-b62b-f7152294ad0b service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:38.428095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-05ffec1e-6b34-4737-b62b-f7152294ad0b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:38.504076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ffec1e-6b34-4737-b62b-f7152294ad0b service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:38.509571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:38.510526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:38.510735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:38.511012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] PATH_INFO: `/v3/users/643cb613baae43b49b0ea37eacb9bf3b` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:38.512220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=643cb613baae43b49b0ea37eacb9bf3b)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:38.514349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:38.734520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '643cb613baae43b49b0ea37eacb9bf3b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:38.735500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '643cb613baae43b49b0ea37eacb9bf3b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:38.745522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '643cb613baae43b49b0ea37eacb9bf3b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:38.761645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '643cb613baae43b49b0ea37eacb9bf3b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:38.836239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '643cb613baae43b49b0ea37eacb9bf3b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:38.843071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1047/2106] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:38 2022] DELETE /identity/v3/users/643cb613baae43b49b0ea37eacb9bf3b => generated 0 bytes in 418 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:38.848645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd692c81-130d-40a5-9d73-55526e8b7c13 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:38.849146 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd692c81-130d-40a5-9d73-55526e8b7c13 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:38.849235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd692c81-130d-40a5-9d73-55526e8b7c13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:38.870803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-dd692c81-130d-40a5-9d73-55526e8b7c13 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:38.872028 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-dd692c81-130d-40a5-9d73-55526e8b7c13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:38.945743 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1060/2107] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:38.955875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:38.957422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:39.018016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:39.033937 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a2894d-9586-4467-a520-608afeccdb4d admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:39.038446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe014b5a-d1a8-421c-8db1-772bb0e94ec9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:39.039202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe014b5a-d1a8-421c-8db1-772bb0e94ec9 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:39.041238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe014b5a-d1a8-421c-8db1-772bb0e94ec9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:39.041238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe014b5a-d1a8-421c-8db1-772bb0e94ec9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:39.050659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe014b5a-d1a8-421c-8db1-772bb0e94ec9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:39.053083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe014b5a-d1a8-421c-8db1-772bb0e94ec9 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:39.066991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1048/2108] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:39.563601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd692c81-130d-40a5-9d73-55526e8b7c13 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:39.566530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-dd692c81-130d-40a5-9d73-55526e8b7c13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:39.582855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd692c81-130d-40a5-9d73-55526e8b7c13 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:39.598408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:39.598408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:39.598408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:39.598408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] PATH_INFO: `/v3/projects/b16c54aef0b74c6f87d551460ee691c9` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:39.602819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=b16c54aef0b74c6f87d551460ee691c9)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:39.604950 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:39.722632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b16c54aef0b74c6f87d551460ee691c9', 'request_id': 'req-c877ea01-aa68-4974-8dec-e9bd2b0056ad'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:39.728819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b16c54aef0b74c6f87d551460ee691c9', 'request_id': 'req-c877ea01-aa68-4974-8dec-e9bd2b0056ad'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:39.937276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b16c54aef0b74c6f87d551460ee691c9', 'request_id': 'req-c877ea01-aa68-4974-8dec-e9bd2b0056ad'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:39.952839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] The token cache is being invalidate because project b16c54aef0b74c6f87d551460ee691c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:39.952839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:39.952839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:39.955417 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1061/2109] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:39 2022] DELETE /identity/v3/projects/b16c54aef0b74c6f87d551460ee691c9 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:39.960798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe014b5a-d1a8-421c-8db1-772bb0e94ec9 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:39.974490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe014b5a-d1a8-421c-8db1-772bb0e94ec9 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:40.044032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:40.044666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:40.044946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:40.045219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] PATH_INFO: `/v3/users/c1d91595ccb74e6b9996f9e00540a127` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:40.046462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1d91595ccb74e6b9996f9e00540a127)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:40.048385 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:40.427759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1d91595ccb74e6b9996f9e00540a127'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:40.428092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1d91595ccb74e6b9996f9e00540a127'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:40.435950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1d91595ccb74e6b9996f9e00540a127'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:40.449567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1d91595ccb74e6b9996f9e00540a127'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:40.527673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1d91595ccb74e6b9996f9e00540a127'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:40.534086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1049/2110] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:39 2022] DELETE /identity/v3/users/c1d91595ccb74e6b9996f9e00540a127 => generated 0 bytes in 574 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:41.130762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:41.135746 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:41.152020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c877ea01-aa68-4974-8dec-e9bd2b0056ad admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:41.227258 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:41.228325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:41.228842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:41.229278 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] PATH_INFO: `/v3/projects/dc8001ce88eb43e19c1363d824896246` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:41.230800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc8001ce88eb43e19c1363d824896246)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:41.233010 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:41.253194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:41.384056 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc8001ce88eb43e19c1363d824896246', 'request_id': 'req-f981e253-d12d-4fd4-8048-102230867954'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:41.391276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc8001ce88eb43e19c1363d824896246', 'request_id': 'req-f981e253-d12d-4fd4-8048-102230867954'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:41.476804 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc8001ce88eb43e19c1363d824896246', 'request_id': 'req-f981e253-d12d-4fd4-8048-102230867954'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:41.496640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] The token cache is being invalidate because project dc8001ce88eb43e19c1363d824896246 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:41.496934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:41.497748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:41.499588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1062/2111] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:41 2022] DELETE /identity/v3/projects/dc8001ce88eb43e19c1363d824896246 => generated 0 bytes in 370 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:41.505120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:41.506684 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:41.521648 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2764e49-e599-4668-9f28-8e1f44f7a642 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:41.606344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20d4994-382e-4f50-aab6-df489f580f9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:41.607054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20d4994-382e-4f50-aab6-df489f580f9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:41.607496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20d4994-382e-4f50-aab6-df489f580f9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:41.607772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20d4994-382e-4f50-aab6-df489f580f9c admin admin] PATH_INFO: `/v3/users/dae7705cc7a34e828f4d4ce5f8583d5e` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:41.608986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20d4994-382e-4f50-aab6-df489f580f9c admin admin] RBAC: Authorizing `identity:delete_user(user_id=dae7705cc7a34e828f4d4ce5f8583d5e)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:41.610966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20d4994-382e-4f50-aab6-df489f580f9c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:41.614336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:41.910988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:41.912509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:41.938435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f981e253-d12d-4fd4-8048-102230867954 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:41.946852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:41.947713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:41.947935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:41.948169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] PATH_INFO: `/v3/users/d53a359ab87b494dae7817665634551c` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:41.949476 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] RBAC: Authorizing `identity:delete_user(user_id=d53a359ab87b494dae7817665634551c)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:41.951501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:42.007548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a20d4994-382e-4f50-aab6-df489f580f9c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dae7705cc7a34e828f4d4ce5f8583d5e'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:42.009462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a20d4994-382e-4f50-aab6-df489f580f9c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dae7705cc7a34e828f4d4ce5f8583d5e'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:42.021446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a20d4994-382e-4f50-aab6-df489f580f9c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dae7705cc7a34e828f4d4ce5f8583d5e'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:42.039415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a20d4994-382e-4f50-aab6-df489f580f9c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dae7705cc7a34e828f4d4ce5f8583d5e'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:42.170073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a20d4994-382e-4f50-aab6-df489f580f9c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dae7705cc7a34e828f4d4ce5f8583d5e'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:42.179258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1050/2112] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:41 2022] DELETE /identity/v3/users/dae7705cc7a34e828f4d4ce5f8583d5e => generated 0 bytes in 676 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:42.356283 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd53a359ab87b494dae7817665634551c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:42.357140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd53a359ab87b494dae7817665634551c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:42.373110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd53a359ab87b494dae7817665634551c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:42.374809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd53a359ab87b494dae7817665634551c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:42.443369 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd53a359ab87b494dae7817665634551c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:42.474698 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1063/2113] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:41 2022] DELETE /identity/v3/users/d53a359ab87b494dae7817665634551c => generated 0 bytes in 564 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:42.492447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b8e3a9-1410-4667-8121-736c34658211 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:42.492995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b8e3a9-1410-4667-8121-736c34658211 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:42.493439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b8e3a9-1410-4667-8121-736c34658211 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:42.526775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-13b8e3a9-1410-4667-8121-736c34658211 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:42.528360 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-13b8e3a9-1410-4667-8121-736c34658211 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:42.610132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1051/2114] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:42.649105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:42.654100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:42.735044 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f4840b-1735-4583-9112-8cee71e2ad4b admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:42.740455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176ad212-3250-4993-be47-0562728c9da8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:42.741076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176ad212-3250-4993-be47-0562728c9da8 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:42.741365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176ad212-3250-4993-be47-0562728c9da8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:42.741620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176ad212-3250-4993-be47-0562728c9da8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:42.751617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176ad212-3250-4993-be47-0562728c9da8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:42.755414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176ad212-3250-4993-be47-0562728c9da8 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:42.766360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1064/2115] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:42.813871 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b8e3a9-1410-4667-8121-736c34658211 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:42.815646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-13b8e3a9-1410-4667-8121-736c34658211 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:42.830900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b8e3a9-1410-4667-8121-736c34658211 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:42.835778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:42.836443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:42.836725 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:42.837001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb admin admin] PATH_INFO: `/v3/projects/547074bf4b2c48258e00d5bf8c174f1e` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:42.838704 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb admin admin] RBAC: Authorizing `identity:delete_project(project_id=547074bf4b2c48258e00d5bf8c174f1e)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:42.843372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:43.038261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '547074bf4b2c48258e00d5bf8c174f1e', 'request_id': 'req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:43.044826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '547074bf4b2c48258e00d5bf8c174f1e', 'request_id': 'req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:43.052628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '547074bf4b2c48258e00d5bf8c174f1e', 'request_id': 'req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:43.162819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb admin admin] The token cache is being invalidate because project 547074bf4b2c48258e00d5bf8c174f1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:43.165647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:43.165647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e7840b6c-d690-45f1-ba26-a79ba28b5bbb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:43.170748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1052/2116] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:42 2022] DELETE /identity/v3/projects/547074bf4b2c48258e00d5bf8c174f1e => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:43.172162 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2529ee-15cd-435d-a9d2-75ebb77685be None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:43.172580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2529ee-15cd-435d-a9d2-75ebb77685be None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:43.172856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2529ee-15cd-435d-a9d2-75ebb77685be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:43.194281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-4a2529ee-15cd-435d-a9d2-75ebb77685be None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:43.195668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4a2529ee-15cd-435d-a9d2-75ebb77685be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:43.205401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:43.262692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1065/2117] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:43.272669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3473a56a-33b9-40b9-a243-a0d76bcd613b None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:43.272951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3473a56a-33b9-40b9-a243-a0d76bcd613b None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:43.273226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3473a56a-33b9-40b9-a243-a0d76bcd613b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:43.282096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:43.293920 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-3473a56a-33b9-40b9-a243-a0d76bcd613b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:43.296018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3473a56a-33b9-40b9-a243-a0d76bcd613b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:43.305377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1053/2118] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:43.310722 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2529ee-15cd-435d-a9d2-75ebb77685be None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:43.320368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3473a56a-33b9-40b9-a243-a0d76bcd613b None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:43.321536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3473a56a-33b9-40b9-a243-a0d76bcd613b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:43.322307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2529ee-15cd-435d-a9d2-75ebb77685be None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:43.350679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28e2c1d-6bd6-430e-99ed-f8c2be401ddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:43.350679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28e2c1d-6bd6-430e-99ed-f8c2be401ddd admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:43.350679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28e2c1d-6bd6-430e-99ed-f8c2be401ddd admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:43.350679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28e2c1d-6bd6-430e-99ed-f8c2be401ddd admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:43.351501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28e2c1d-6bd6-430e-99ed-f8c2be401ddd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:50:43.351501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28e2c1d-6bd6-430e-99ed-f8c2be401ddd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:43.351501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28e2c1d-6bd6-430e-99ed-f8c2be401ddd admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:43.352115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3473a56a-33b9-40b9-a243-a0d76bcd613b None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:43.352115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:43.352115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:43.352115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:43.352633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] PATH_INFO: `/v3/projects/520b762c22a44f1485ce89b426de7a57` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:43.352633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] RBAC: Authorizing `identity:delete_project(project_id=520b762c22a44f1485ce89b426de7a57)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:43.352633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:43.352928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1066/2119] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:50:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:50:43.352928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28e2c1d-6bd6-430e-99ed-f8c2be401ddd admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:43.363853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28e2c1d-6bd6-430e-99ed-f8c2be401ddd admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:43.369381 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65ce9df-e816-4e9c-9536-aee704fa30cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:43.370024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65ce9df-e816-4e9c-9536-aee704fa30cc admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:43.370334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65ce9df-e816-4e9c-9536-aee704fa30cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:43.370609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65ce9df-e816-4e9c-9536-aee704fa30cc admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:43.371587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65ce9df-e816-4e9c-9536-aee704fa30cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:43.373163 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65ce9df-e816-4e9c-9536-aee704fa30cc admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:43.522321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '520b762c22a44f1485ce89b426de7a57', 'request_id': 'req-6bcf5ad4-1980-4397-a919-4fdff0168723'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:43.528610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '520b762c22a44f1485ce89b426de7a57', 'request_id': 'req-6bcf5ad4-1980-4397-a919-4fdff0168723'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:43.537831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '520b762c22a44f1485ce89b426de7a57', 'request_id': 'req-6bcf5ad4-1980-4397-a919-4fdff0168723'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:43.572592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1067/2120] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:43 2022] POST /identity/v3/projects => generated 373 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:50:43.575737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65ce9df-e816-4e9c-9536-aee704fa30cc admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:43.631506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] The token cache is being invalidate because project 520b762c22a44f1485ce89b426de7a57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:43.632174 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:43.635196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:43.635441 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1054/2121] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:43 2022] DELETE /identity/v3/projects/520b762c22a44f1485ce89b426de7a57 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:43.639273 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:43.640604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:43.646854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65ce9df-e816-4e9c-9536-aee704fa30cc admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:43.652621 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8e4672-534c-43dc-964d-f0290a5e0f3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:43.653394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8e4672-534c-43dc-964d-f0290a5e0f3f admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:43.653667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8e4672-534c-43dc-964d-f0290a5e0f3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:43.653865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8e4672-534c-43dc-964d-f0290a5e0f3f admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:43.654658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8e4672-534c-43dc-964d-f0290a5e0f3f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:43.656831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8e4672-534c-43dc-964d-f0290a5e0f3f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:43.711972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcf5ad4-1980-4397-a919-4fdff0168723 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:43.716891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:43.717521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:43.717802 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:43.718075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] PATH_INFO: `/v3/users/4a4ea601eae144f08ab44953935279b3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:43.719298 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a4ea601eae144f08ab44953935279b3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:43.721278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:43.829335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1068/2122] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:43 2022] POST /identity/v3/users => generated 287 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:50:43.835115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8e4672-534c-43dc-964d-f0290a5e0f3f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:43.839394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9d8e4672-534c-43dc-964d-f0290a5e0f3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:43.863241 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8e4672-534c-43dc-964d-f0290a5e0f3f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:43.870292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18859b3a-99f0-4831-8efd-18d53d877361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:43.870989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18859b3a-99f0-4831-8efd-18d53d877361 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:43.872628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18859b3a-99f0-4831-8efd-18d53d877361 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:43.872628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18859b3a-99f0-4831-8efd-18d53d877361 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:43.872628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18859b3a-99f0-4831-8efd-18d53d877361 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:43.874086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18859b3a-99f0-4831-8efd-18d53d877361 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:43.890648 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1069/2123] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:43.891999 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18859b3a-99f0-4831-8efd-18d53d877361 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:43.903172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18859b3a-99f0-4831-8efd-18d53d877361 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:43.907898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3aee09-a762-45d1-adb1-fd536e2f8bef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:43.908928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3aee09-a762-45d1-adb1-fd536e2f8bef admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:43.908928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3aee09-a762-45d1-adb1-fd536e2f8bef admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:43.909098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3aee09-a762-45d1-adb1-fd536e2f8bef admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:43.909899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3aee09-a762-45d1-adb1-fd536e2f8bef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:43.911542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3aee09-a762-45d1-adb1-fd536e2f8bef admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:43.922140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1070/2124] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:43.926384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3aee09-a762-45d1-adb1-fd536e2f8bef admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:43.937826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3aee09-a762-45d1-adb1-fd536e2f8bef admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:43.944740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b917279b-74c5-44e1-9f8e-7390db06cbec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:43.945405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b917279b-74c5-44e1-9f8e-7390db06cbec admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:43.945685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b917279b-74c5-44e1-9f8e-7390db06cbec admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:43.945958 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b917279b-74c5-44e1-9f8e-7390db06cbec admin admin] PATH_INFO: `/v3/projects/3cf3b9526fc541048f921613ded8ef2d/users/ae04cdfc35964cf58502cefd1cc912e9/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:43.963885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b917279b-74c5-44e1-9f8e-7390db06cbec admin admin] RBAC: Authorizing `identity:create_grant(project_id=3cf3b9526fc541048f921613ded8ef2d, user_id=ae04cdfc35964cf58502cefd1cc912e9, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:43.966670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b917279b-74c5-44e1-9f8e-7390db06cbec admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:44.071382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1071/2125] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:43 2022] PUT /identity/v3/projects/3cf3b9526fc541048f921613ded8ef2d/users/ae04cdfc35964cf58502cefd1cc912e9/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:44.072656 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a4ea601eae144f08ab44953935279b3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:44.073279 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a4ea601eae144f08ab44953935279b3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:44.077128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b97f75-d258-4458-8d0a-4157e699a10e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:44.077363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b97f75-d258-4458-8d0a-4157e699a10e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:44.077597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b97f75-d258-4458-8d0a-4157e699a10e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:44.081479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a4ea601eae144f08ab44953935279b3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:44.098404 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a4ea601eae144f08ab44953935279b3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:44.103884 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e3b97f75-d258-4458-8d0a-4157e699a10e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:44.103884 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e3b97f75-d258-4458-8d0a-4157e699a10e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:44.164938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1072/2126] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:44.177545 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b97f75-d258-4458-8d0a-4157e699a10e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:44.179179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e3b97f75-d258-4458-8d0a-4157e699a10e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:44.190205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a4ea601eae144f08ab44953935279b3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:44.199422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1055/2127] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:43 2022] DELETE /identity/v3/users/4a4ea601eae144f08ab44953935279b3 => generated 0 bytes in 560 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:44.258878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b97f75-d258-4458-8d0a-4157e699a10e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:44.310479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0c5ea6-fc23-44c2-9375-f3b1e4180c2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:44.311095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0c5ea6-fc23-44c2-9375-f3b1e4180c2c service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:44.311353 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0c5ea6-fc23-44c2-9375-f3b1e4180c2c service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:44.311691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0c5ea6-fc23-44c2-9375-f3b1e4180c2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:44.313737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1f0c5ea6-fc23-44c2-9375-f3b1e4180c2c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:44.381815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0c5ea6-fc23-44c2-9375-f3b1e4180c2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:44.383411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0c5ea6-fc23-44c2-9375-f3b1e4180c2c service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:44.394542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1073/2128] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:50:44.734443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:44.734443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:44.747589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffba8d1e-c0e3-4405-a8dc-ee633cc280c3 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:44.752808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8219f35-8060-432f-b32e-7bf1181d7ca4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:44.753625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8219f35-8060-432f-b32e-7bf1181d7ca4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:44.754118 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8219f35-8060-432f-b32e-7bf1181d7ca4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:44.754618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8219f35-8060-432f-b32e-7bf1181d7ca4 admin admin] PATH_INFO: `/v3/projects/7f19139273854430a865796391971664` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:44.756067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8219f35-8060-432f-b32e-7bf1181d7ca4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f19139273854430a865796391971664)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:44.758442 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8219f35-8060-432f-b32e-7bf1181d7ca4 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:44.990998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a8219f35-8060-432f-b32e-7bf1181d7ca4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f19139273854430a865796391971664', 'request_id': 'req-a8219f35-8060-432f-b32e-7bf1181d7ca4'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:44.997412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a8219f35-8060-432f-b32e-7bf1181d7ca4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f19139273854430a865796391971664', 'request_id': 'req-a8219f35-8060-432f-b32e-7bf1181d7ca4'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:45.006065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a8219f35-8060-432f-b32e-7bf1181d7ca4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f19139273854430a865796391971664', 'request_id': 'req-a8219f35-8060-432f-b32e-7bf1181d7ca4'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:45.030934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:45.116505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a8219f35-8060-432f-b32e-7bf1181d7ca4 admin admin] The token cache is being invalidate because project 7f19139273854430a865796391971664 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:45.129424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a8219f35-8060-432f-b32e-7bf1181d7ca4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:45.129424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a8219f35-8060-432f-b32e-7bf1181d7ca4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:45.129424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1056/2129] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:44 2022] DELETE /identity/v3/projects/7f19139273854430a865796391971664 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:45.130844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c39cdf7-b52a-4fa7-8b31-cfb437b657c5 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:45.130844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c39cdf7-b52a-4fa7-8b31-cfb437b657c5 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:45.130844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c39cdf7-b52a-4fa7-8b31-cfb437b657c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:45.149705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-9c39cdf7-b52a-4fa7-8b31-cfb437b657c5 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:45.151014 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9c39cdf7-b52a-4fa7-8b31-cfb437b657c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:45.248101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:45.261264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1074/2130] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:45.269530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0134ba-b79d-4780-aaca-d406b18c9769 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:45.269817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0134ba-b79d-4780-aaca-d406b18c9769 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:45.270096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0134ba-b79d-4780-aaca-d406b18c9769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:45.293713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-eb0134ba-b79d-4780-aaca-d406b18c9769 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:45.295412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-eb0134ba-b79d-4780-aaca-d406b18c9769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:45.305023 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1057/2131] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:45.309427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c39cdf7-b52a-4fa7-8b31-cfb437b657c5 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:45.320068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c39cdf7-b52a-4fa7-8b31-cfb437b657c5 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:45.325061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd421f3f-dcda-425c-b1b3-e3ac7fe1d378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:45.325707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd421f3f-dcda-425c-b1b3-e3ac7fe1d378 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:45.327530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd421f3f-dcda-425c-b1b3-e3ac7fe1d378 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:45.327530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd421f3f-dcda-425c-b1b3-e3ac7fe1d378 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:45.327530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd421f3f-dcda-425c-b1b3-e3ac7fe1d378 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:50:45.327530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd421f3f-dcda-425c-b1b3-e3ac7fe1d378 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:45.329055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd421f3f-dcda-425c-b1b3-e3ac7fe1d378 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:45.346487 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1075/2132] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:50:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:50:45.349736 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0134ba-b79d-4780-aaca-d406b18c9769 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:45.362546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0134ba-b79d-4780-aaca-d406b18c9769 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:45.369761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1603419-e7b3-4fea-8286-3afc9255f5ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:45.375095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1603419-e7b3-4fea-8286-3afc9255f5ef admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:45.375576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1603419-e7b3-4fea-8286-3afc9255f5ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:45.377087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1603419-e7b3-4fea-8286-3afc9255f5ef admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:45.378551 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1603419-e7b3-4fea-8286-3afc9255f5ef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:45.381089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1603419-e7b3-4fea-8286-3afc9255f5ef admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:45.484914 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1058/2133] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:45 2022] POST /identity/v3/projects => generated 381 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:45.490662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd421f3f-dcda-425c-b1b3-e3ac7fe1d378 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:45.511858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd421f3f-dcda-425c-b1b3-e3ac7fe1d378 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:45.582596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d151b4-a58b-4cbe-b523-c6b2b7a6b7f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:45.583116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d151b4-a58b-4cbe-b523-c6b2b7a6b7f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:45.583394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d151b4-a58b-4cbe-b523-c6b2b7a6b7f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:45.583661 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d151b4-a58b-4cbe-b523-c6b2b7a6b7f0 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:45.584329 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d151b4-a58b-4cbe-b523-c6b2b7a6b7f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:45.586293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d151b4-a58b-4cbe-b523-c6b2b7a6b7f0 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:45.669388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1076/2134] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:45 2022] POST /identity/v3/users => generated 291 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:45.670721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1603419-e7b3-4fea-8286-3afc9255f5ef admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:45.687988 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1603419-e7b3-4fea-8286-3afc9255f5ef admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:45.696289 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6786e941-9a62-4405-beb5-874fc204df19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:45.696963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6786e941-9a62-4405-beb5-874fc204df19 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:45.697231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6786e941-9a62-4405-beb5-874fc204df19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:45.697673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6786e941-9a62-4405-beb5-874fc204df19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:45.698387 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6786e941-9a62-4405-beb5-874fc204df19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:45.700010 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6786e941-9a62-4405-beb5-874fc204df19 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:45.718425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1059/2135] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:45 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:45.721368 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d151b4-a58b-4cbe-b523-c6b2b7a6b7f0 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:45.734124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d151b4-a58b-4cbe-b523-c6b2b7a6b7f0 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:45.741713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe085c71-35fd-419d-80ae-478c8e0ace1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:45.742542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe085c71-35fd-419d-80ae-478c8e0ace1c admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:45.743004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe085c71-35fd-419d-80ae-478c8e0ace1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:45.743284 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe085c71-35fd-419d-80ae-478c8e0ace1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:45.743946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe085c71-35fd-419d-80ae-478c8e0ace1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:45.745647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe085c71-35fd-419d-80ae-478c8e0ace1c admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:45.759827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1077/2136] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:45 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:45.764180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6786e941-9a62-4405-beb5-874fc204df19 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:45.774968 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6786e941-9a62-4405-beb5-874fc204df19 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:45.780061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b31b615-0fa5-4939-94a7-38875b67373e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:45.780720 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b31b615-0fa5-4939-94a7-38875b67373e admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:45.781007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b31b615-0fa5-4939-94a7-38875b67373e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:45.781452 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b31b615-0fa5-4939-94a7-38875b67373e admin admin] PATH_INFO: `/v3/projects/472b6ecbf5d942cba36d760f36d5d545/users/4b9437ca28624dde97c8df19d3068d4b/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:45.801088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b31b615-0fa5-4939-94a7-38875b67373e admin admin] RBAC: Authorizing `identity:create_grant(project_id=472b6ecbf5d942cba36d760f36d5d545, user_id=4b9437ca28624dde97c8df19d3068d4b, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:45.803823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b31b615-0fa5-4939-94a7-38875b67373e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:45.910828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1060/2137] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:45 2022] PUT /identity/v3/projects/472b6ecbf5d942cba36d760f36d5d545/users/4b9437ca28624dde97c8df19d3068d4b/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:45.923577 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78247f11-b10b-4117-8b40-712cb3c382a6 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:45.923856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78247f11-b10b-4117-8b40-712cb3c382a6 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:45.924130 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78247f11-b10b-4117-8b40-712cb3c382a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:45.968013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-78247f11-b10b-4117-8b40-712cb3c382a6 None None] MFA Rules not processed for user `4b9437ca28624dde97c8df19d3068d4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:45.969673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-78247f11-b10b-4117-8b40-712cb3c382a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:46.029352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da120194-6c4a-4b0a-ae6f-e23ee8f55a68 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:46.030074 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da120194-6c4a-4b0a-ae6f-e23ee8f55a68 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:46.030619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da120194-6c4a-4b0a-ae6f-e23ee8f55a68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:46.058313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1078/2138] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:45 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:46.061571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78247f11-b10b-4117-8b40-712cb3c382a6 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:46.068566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-da120194-6c4a-4b0a-ae6f-e23ee8f55a68 None None] MFA Rules not processed for user `ae04cdfc35964cf58502cefd1cc912e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:46.069940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-da120194-6c4a-4b0a-ae6f-e23ee8f55a68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:46.155870 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:46.159879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1061/2139] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:46 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:46.163017 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78247f11-b10b-4117-8b40-712cb3c382a6 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:46.180110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65344d5-446b-4e45-abc1-bd57eeb164a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:46.180848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65344d5-446b-4e45-abc1-bd57eeb164a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:46.181120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65344d5-446b-4e45-abc1-bd57eeb164a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:46.181384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65344d5-446b-4e45-abc1-bd57eeb164a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:46.182073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65344d5-446b-4e45-abc1-bd57eeb164a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:46.184143 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65344d5-446b-4e45-abc1-bd57eeb164a0 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:46.192139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da120194-6c4a-4b0a-ae6f-e23ee8f55a68 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:46.193635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-da120194-6c4a-4b0a-ae6f-e23ee8f55a68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:46.265633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da120194-6c4a-4b0a-ae6f-e23ee8f55a68 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:46.270525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f600f4a-13cf-4521-b3e3-9d6b4d5932b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:46.271392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f600f4a-13cf-4521-b3e3-9d6b4d5932b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:46.271695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f600f4a-13cf-4521-b3e3-9d6b4d5932b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:46.271959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f600f4a-13cf-4521-b3e3-9d6b4d5932b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:46.281526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f600f4a-13cf-4521-b3e3-9d6b4d5932b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:46.283005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f600f4a-13cf-4521-b3e3-9d6b4d5932b0 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:46.293679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1062/2140] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:46 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:46.309484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1079/2141] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:46 2022] POST /identity/v3/projects => generated 379 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:46.315021 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f600f4a-13cf-4521-b3e3-9d6b4d5932b0 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:46.333853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f600f4a-13cf-4521-b3e3-9d6b4d5932b0 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:46.426830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb095bf5-599e-483c-804f-4ca90acdb7de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:46.429253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb095bf5-599e-483c-804f-4ca90acdb7de admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:46.429253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb095bf5-599e-483c-804f-4ca90acdb7de admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:46.429253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb095bf5-599e-483c-804f-4ca90acdb7de admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:46.429253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb095bf5-599e-483c-804f-4ca90acdb7de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:46.430835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb095bf5-599e-483c-804f-4ca90acdb7de admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:46.580144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1063/2142] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:46 2022] POST /identity/v3/users => generated 296 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:46.580641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65344d5-446b-4e45-abc1-bd57eeb164a0 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:46.593540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65344d5-446b-4e45-abc1-bd57eeb164a0 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:46.599209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa65ee1-3878-4a16-88d4-afdf8247b914 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:46.599808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa65ee1-3878-4a16-88d4-afdf8247b914 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:46.600090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa65ee1-3878-4a16-88d4-afdf8247b914 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:46.600363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa65ee1-3878-4a16-88d4-afdf8247b914 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:46.601031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa65ee1-3878-4a16-88d4-afdf8247b914 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:46.602764 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa65ee1-3878-4a16-88d4-afdf8247b914 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:46.614529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1080/2143] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:46 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:50:46.619133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb095bf5-599e-483c-804f-4ca90acdb7de admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:46.630800 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb095bf5-599e-483c-804f-4ca90acdb7de admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:46.636773 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae0db36-ce5f-408f-b73a-ca458542afe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:46.637527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae0db36-ce5f-408f-b73a-ca458542afe5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:46.637856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae0db36-ce5f-408f-b73a-ca458542afe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:46.638304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae0db36-ce5f-408f-b73a-ca458542afe5 admin admin] PATH_INFO: `/v3/projects/4ceccf74118848b7ae96096e015fd8ef/users/c02ab8b7664843cba0a55af9203cec66/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:46.658927 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae0db36-ce5f-408f-b73a-ca458542afe5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ceccf74118848b7ae96096e015fd8ef, user_id=c02ab8b7664843cba0a55af9203cec66, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:46.662146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae0db36-ce5f-408f-b73a-ca458542afe5 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:46.843375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1064/2144] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:46 2022] PUT /identity/v3/projects/4ceccf74118848b7ae96096e015fd8ef/users/c02ab8b7664843cba0a55af9203cec66/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:46.844074 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa65ee1-3878-4a16-88d4-afdf8247b914 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:46.855274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa65ee1-3878-4a16-88d4-afdf8247b914 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:46.923975 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d3b37f-2296-4bfb-abe9-a114e50dfc62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:46.924703 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d3b37f-2296-4bfb-abe9-a114e50dfc62 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:46.925234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d3b37f-2296-4bfb-abe9-a114e50dfc62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:46.925515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d3b37f-2296-4bfb-abe9-a114e50dfc62 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:46.930921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d3b37f-2296-4bfb-abe9-a114e50dfc62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:46.930921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d3b37f-2296-4bfb-abe9-a114e50dfc62 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:46.938145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1081/2145] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:46 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:46.943596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae0db36-ce5f-408f-b73a-ca458542afe5 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:46.954304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae0db36-ce5f-408f-b73a-ca458542afe5 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:46.960108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33976189-01e2-4017-80ff-22ef231aa55f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:46.960884 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33976189-01e2-4017-80ff-22ef231aa55f admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:46.961247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33976189-01e2-4017-80ff-22ef231aa55f admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:46.961524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33976189-01e2-4017-80ff-22ef231aa55f admin admin] PATH_INFO: `/v3/projects/4ceccf74118848b7ae96096e015fd8ef/users/c02ab8b7664843cba0a55af9203cec66/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:46.963567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33976189-01e2-4017-80ff-22ef231aa55f admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ceccf74118848b7ae96096e015fd8ef, user_id=c02ab8b7664843cba0a55af9203cec66, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:46.966299 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33976189-01e2-4017-80ff-22ef231aa55f admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:46.977127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1065/2146] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:46 2022] PUT /identity/v3/projects/4ceccf74118848b7ae96096e015fd8ef/users/c02ab8b7664843cba0a55af9203cec66/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:46.988194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa08f9d-1f46-44bc-b9ab-1cddd64cd001 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:46.988469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa08f9d-1f46-44bc-b9ab-1cddd64cd001 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:46.988743 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa08f9d-1f46-44bc-b9ab-1cddd64cd001 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:47.016982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-efa08f9d-1f46-44bc-b9ab-1cddd64cd001 None None] MFA Rules not processed for user `c02ab8b7664843cba0a55af9203cec66`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:47.018245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-efa08f9d-1f46-44bc-b9ab-1cddd64cd001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:47.083407 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:47.093481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1082/2147] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:46 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:47.309244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:47.520085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33976189-01e2-4017-80ff-22ef231aa55f admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:47.520085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-33976189-01e2-4017-80ff-22ef231aa55f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:47.595286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33976189-01e2-4017-80ff-22ef231aa55f admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:47.601142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091526db-2049-449f-b629-4430a3222044 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:47.601775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091526db-2049-449f-b629-4430a3222044 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:47.602053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091526db-2049-449f-b629-4430a3222044 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:47.602410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091526db-2049-449f-b629-4430a3222044 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:47.615149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091526db-2049-449f-b629-4430a3222044 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:47.616928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091526db-2049-449f-b629-4430a3222044 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:47.627572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1066/2148] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:47 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:50:48.278447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa08f9d-1f46-44bc-b9ab-1cddd64cd001 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:48.303436 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:48.348599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa08f9d-1f46-44bc-b9ab-1cddd64cd001 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:48.353075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:48.358404 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:48.358404 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:48.358404 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] PATH_INFO: `/v3/users/4b9437ca28624dde97c8df19d3068d4b` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:48.358404 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b9437ca28624dde97c8df19d3068d4b)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:48.358404 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:48.559683 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b9437ca28624dde97c8df19d3068d4b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:48.560409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b9437ca28624dde97c8df19d3068d4b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:48.567819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b9437ca28624dde97c8df19d3068d4b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:48.568551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b9437ca28624dde97c8df19d3068d4b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:48.681737 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b9437ca28624dde97c8df19d3068d4b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:48.702193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1083/2149] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:48 2022] DELETE /identity/v3/users/4b9437ca28624dde97c8df19d3068d4b => generated 0 bytes in 424 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:48.707185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bdbb85-f9e2-4f98-b8f2-a4b97781982c None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:48.707709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bdbb85-f9e2-4f98-b8f2-a4b97781982c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:48.707857 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bdbb85-f9e2-4f98-b8f2-a4b97781982c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:48.721338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:48.727893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-91bdbb85-f9e2-4f98-b8f2-a4b97781982c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:48.729119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-91bdbb85-f9e2-4f98-b8f2-a4b97781982c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:48.791177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1067/2150] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:48.802808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:48.804297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:48.870215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416e7d1e-9786-4d1f-bfd5-9bdf6d994bd1 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:48.873807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882f7cd8-0607-4745-a897-25c77df66ed9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:48.874443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882f7cd8-0607-4745-a897-25c77df66ed9 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:48.874722 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882f7cd8-0607-4745-a897-25c77df66ed9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:48.875000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882f7cd8-0607-4745-a897-25c77df66ed9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:48.884672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882f7cd8-0607-4745-a897-25c77df66ed9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:48.886115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882f7cd8-0607-4745-a897-25c77df66ed9 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:48.900767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1084/2151] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:50:49.403001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bdbb85-f9e2-4f98-b8f2-a4b97781982c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:49.404590 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-91bdbb85-f9e2-4f98-b8f2-a4b97781982c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:49.423707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bdbb85-f9e2-4f98-b8f2-a4b97781982c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:49.432638 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7624bc56-8bb6-42a9-a89b-02c843059a22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:49.433701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7624bc56-8bb6-42a9-a89b-02c843059a22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:49.434222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7624bc56-8bb6-42a9-a89b-02c843059a22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:49.434770 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7624bc56-8bb6-42a9-a89b-02c843059a22 admin admin] PATH_INFO: `/v3/projects/472b6ecbf5d942cba36d760f36d5d545` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:49.436738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7624bc56-8bb6-42a9-a89b-02c843059a22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=472b6ecbf5d942cba36d760f36d5d545)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:49.441109 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7624bc56-8bb6-42a9-a89b-02c843059a22 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:49.572424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-7624bc56-8bb6-42a9-a89b-02c843059a22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '472b6ecbf5d942cba36d760f36d5d545', 'request_id': 'req-7624bc56-8bb6-42a9-a89b-02c843059a22'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:49.579908 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-7624bc56-8bb6-42a9-a89b-02c843059a22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '472b6ecbf5d942cba36d760f36d5d545', 'request_id': 'req-7624bc56-8bb6-42a9-a89b-02c843059a22'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:49.586954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-7624bc56-8bb6-42a9-a89b-02c843059a22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '472b6ecbf5d942cba36d760f36d5d545', 'request_id': 'req-7624bc56-8bb6-42a9-a89b-02c843059a22'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:49.668049 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-7624bc56-8bb6-42a9-a89b-02c843059a22 admin admin] The token cache is being invalidate because project 472b6ecbf5d942cba36d760f36d5d545 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:49.668336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-7624bc56-8bb6-42a9-a89b-02c843059a22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:49.668976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-7624bc56-8bb6-42a9-a89b-02c843059a22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:49.672124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1068/2152] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:49 2022] DELETE /identity/v3/projects/472b6ecbf5d942cba36d760f36d5d545 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:49.674957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882f7cd8-0607-4745-a897-25c77df66ed9 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:49.691002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882f7cd8-0607-4745-a897-25c77df66ed9 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:49.748109 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d7f4e3-41cc-4c2d-9b32-bf9abb7aab38 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:49.748965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d7f4e3-41cc-4c2d-9b32-bf9abb7aab38 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:49.749429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d7f4e3-41cc-4c2d-9b32-bf9abb7aab38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:49.758466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:49.786679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-80d7f4e3-41cc-4c2d-9b32-bf9abb7aab38 None None] MFA Rules not processed for user `ae04cdfc35964cf58502cefd1cc912e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:49.788299 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a873ea3-a165-42a8-995a-f23ef54edf0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:49.788812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-80d7f4e3-41cc-4c2d-9b32-bf9abb7aab38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:49.788964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a873ea3-a165-42a8-995a-f23ef54edf0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:49.789226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a873ea3-a165-42a8-995a-f23ef54edf0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:49.789501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a873ea3-a165-42a8-995a-f23ef54edf0a admin admin] PATH_INFO: `/v3/users/c02ab8b7664843cba0a55af9203cec66` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:49.790713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a873ea3-a165-42a8-995a-f23ef54edf0a admin admin] RBAC: Authorizing `identity:delete_user(user_id=c02ab8b7664843cba0a55af9203cec66)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:49.793095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a873ea3-a165-42a8-995a-f23ef54edf0a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:49.857484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1069/2153] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:49 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:50.033419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4a873ea3-a165-42a8-995a-f23ef54edf0a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c02ab8b7664843cba0a55af9203cec66'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:50.034868 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4a873ea3-a165-42a8-995a-f23ef54edf0a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c02ab8b7664843cba0a55af9203cec66'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:50.043286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4a873ea3-a165-42a8-995a-f23ef54edf0a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c02ab8b7664843cba0a55af9203cec66'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:50.044020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4a873ea3-a165-42a8-995a-f23ef54edf0a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c02ab8b7664843cba0a55af9203cec66'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:50.127767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4a873ea3-a165-42a8-995a-f23ef54edf0a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c02ab8b7664843cba0a55af9203cec66'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:50.161321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1085/2154] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:50:49 2022] DELETE /identity/v3/users/c02ab8b7664843cba0a55af9203cec66 => generated 0 bytes in 487 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:50.718450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d7f4e3-41cc-4c2d-9b32-bf9abb7aab38 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:50.719945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-80d7f4e3-41cc-4c2d-9b32-bf9abb7aab38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:50.736460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d7f4e3-41cc-4c2d-9b32-bf9abb7aab38 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:50.812796 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:50.845040 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a3ebd0-ee2e-405e-9fc6-c058d0013799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:50.845715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a3ebd0-ee2e-405e-9fc6-c058d0013799 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:50.849139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a3ebd0-ee2e-405e-9fc6-c058d0013799 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:50.849139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a3ebd0-ee2e-405e-9fc6-c058d0013799 admin admin] PATH_INFO: `/v3/projects/4ceccf74118848b7ae96096e015fd8ef` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:50.849139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a3ebd0-ee2e-405e-9fc6-c058d0013799 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ceccf74118848b7ae96096e015fd8ef)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:50.849791 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a3ebd0-ee2e-405e-9fc6-c058d0013799 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:51.012973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-01a3ebd0-ee2e-405e-9fc6-c058d0013799 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ceccf74118848b7ae96096e015fd8ef', 'request_id': 'req-01a3ebd0-ee2e-405e-9fc6-c058d0013799'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:51.020619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-01a3ebd0-ee2e-405e-9fc6-c058d0013799 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ceccf74118848b7ae96096e015fd8ef', 'request_id': 'req-01a3ebd0-ee2e-405e-9fc6-c058d0013799'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:51.030966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-01a3ebd0-ee2e-405e-9fc6-c058d0013799 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ceccf74118848b7ae96096e015fd8ef', 'request_id': 'req-01a3ebd0-ee2e-405e-9fc6-c058d0013799'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:51.162458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-01a3ebd0-ee2e-405e-9fc6-c058d0013799 admin admin] The token cache is being invalidate because project 4ceccf74118848b7ae96096e015fd8ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:50:51.162740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-01a3ebd0-ee2e-405e-9fc6-c058d0013799 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:51.163410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-01a3ebd0-ee2e-405e-9fc6-c058d0013799 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:50:51.164965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1070/2155] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:50:50 2022] DELETE /identity/v3/projects/4ceccf74118848b7ae96096e015fd8ef => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:50:51.169864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee5f9cc-7232-4a8b-84f2-6b55fc885d95 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:51.170397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee5f9cc-7232-4a8b-84f2-6b55fc885d95 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:51.170510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee5f9cc-7232-4a8b-84f2-6b55fc885d95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:51.191303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5ee5f9cc-7232-4a8b-84f2-6b55fc885d95 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:51.192524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5ee5f9cc-7232-4a8b-84f2-6b55fc885d95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:51.259844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1086/2156] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:51.269559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4ff498-1325-4e66-b468-e83e3e149f1d None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:51.269976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4ff498-1325-4e66-b468-e83e3e149f1d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:51.270422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4ff498-1325-4e66-b468-e83e3e149f1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:51.291313 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-be4ff498-1325-4e66-b468-e83e3e149f1d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:51.292541 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-be4ff498-1325-4e66-b468-e83e3e149f1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:51.301166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1071/2157] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:50:51.305666 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee5f9cc-7232-4a8b-84f2-6b55fc885d95 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:51.316660 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee5f9cc-7232-4a8b-84f2-6b55fc885d95 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:51.321550 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c464c5-89d5-4b1b-9a35-fa626dacd420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:51.324919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c464c5-89d5-4b1b-9a35-fa626dacd420 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:51.324919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c464c5-89d5-4b1b-9a35-fa626dacd420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:51.324919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c464c5-89d5-4b1b-9a35-fa626dacd420 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:51.324919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c464c5-89d5-4b1b-9a35-fa626dacd420 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:50:51.324919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c464c5-89d5-4b1b-9a35-fa626dacd420 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:51.325327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c464c5-89d5-4b1b-9a35-fa626dacd420 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:51.338654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1087/2158] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:50:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:50:51.343046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4ff498-1325-4e66-b468-e83e3e149f1d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:51.353188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4ff498-1325-4e66-b468-e83e3e149f1d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:51.365243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2f4839-600f-4cc2-b051-7deddfa74002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:51.365243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2f4839-600f-4cc2-b051-7deddfa74002 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:51.365243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2f4839-600f-4cc2-b051-7deddfa74002 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:51.365243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2f4839-600f-4cc2-b051-7deddfa74002 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:51.365609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2f4839-600f-4cc2-b051-7deddfa74002 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:51.365609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2f4839-600f-4cc2-b051-7deddfa74002 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:51.505089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1072/2159] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:50:51 2022] POST /identity/v3/projects => generated 383 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:50:51.507173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c464c5-89d5-4b1b-9a35-fa626dacd420 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:51.519876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c464c5-89d5-4b1b-9a35-fa626dacd420 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:51.569349 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:50:51.581679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae45ab2-561f-4bce-9943-f3ec6e0e5885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:51.582327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae45ab2-561f-4bce-9943-f3ec6e0e5885 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:51.582625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae45ab2-561f-4bce-9943-f3ec6e0e5885 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:51.582900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae45ab2-561f-4bce-9943-f3ec6e0e5885 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:51.583620 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae45ab2-561f-4bce-9943-f3ec6e0e5885 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:51.585593 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae45ab2-561f-4bce-9943-f3ec6e0e5885 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:51.740849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1088/2160] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:50:51 2022] POST /identity/v3/users => generated 292 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:50:51.745528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2f4839-600f-4cc2-b051-7deddfa74002 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:51.756885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2f4839-600f-4cc2-b051-7deddfa74002 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:51.761955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd941a6-823f-47c4-ba6f-c885bc5fbda8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:51.762687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd941a6-823f-47c4-ba6f-c885bc5fbda8 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:51.763043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd941a6-823f-47c4-ba6f-c885bc5fbda8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:51.763392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd941a6-823f-47c4-ba6f-c885bc5fbda8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:51.764124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd941a6-823f-47c4-ba6f-c885bc5fbda8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:51.765811 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd941a6-823f-47c4-ba6f-c885bc5fbda8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:51.777042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1073/2161] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:51.780883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae45ab2-561f-4bce-9943-f3ec6e0e5885 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:51.791323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae45ab2-561f-4bce-9943-f3ec6e0e5885 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:51.796170 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f10d0f6-07a7-4070-b7f9-92bbb0fcd47b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:51.796779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f10d0f6-07a7-4070-b7f9-92bbb0fcd47b admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:51.797057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f10d0f6-07a7-4070-b7f9-92bbb0fcd47b admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:51.797328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f10d0f6-07a7-4070-b7f9-92bbb0fcd47b admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:51.798005 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f10d0f6-07a7-4070-b7f9-92bbb0fcd47b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:51.800268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f10d0f6-07a7-4070-b7f9-92bbb0fcd47b admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:51.812241 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1089/2162] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:50:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:50:51.817268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd941a6-823f-47c4-ba6f-c885bc5fbda8 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:51.828033 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd941a6-823f-47c4-ba6f-c885bc5fbda8 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:51.833089 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18af417-3549-432c-bbfe-e65a584ddc3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:51.833800 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18af417-3549-432c-bbfe-e65a584ddc3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:51.836066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18af417-3549-432c-bbfe-e65a584ddc3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:51.836066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18af417-3549-432c-bbfe-e65a584ddc3c admin admin] PATH_INFO: `/v3/projects/daac215191a944abb73c2e297241f09e/users/51447f0852584a47b59894ee8a130041/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:51.851219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18af417-3549-432c-bbfe-e65a584ddc3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=daac215191a944abb73c2e297241f09e, user_id=51447f0852584a47b59894ee8a130041, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:51.853971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18af417-3549-432c-bbfe-e65a584ddc3c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:51.958469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1074/2163] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:50:51 2022] PUT /identity/v3/projects/daac215191a944abb73c2e297241f09e/users/51447f0852584a47b59894ee8a130041/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:50:51.969602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdc3777-70c4-4d82-86d7-46f1addea9b5 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:51.969885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdc3777-70c4-4d82-86d7-46f1addea9b5 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:51.970180 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdc3777-70c4-4d82-86d7-46f1addea9b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:52.001082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-6fdc3777-70c4-4d82-86d7-46f1addea9b5 None None] MFA Rules not processed for user `51447f0852584a47b59894ee8a130041`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:50:52.002353 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6fdc3777-70c4-4d82-86d7-46f1addea9b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:52.070508 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1090/2164] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:50:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:50:52.084243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18af417-3549-432c-bbfe-e65a584ddc3c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:50:52.086066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a18af417-3549-432c-bbfe-e65a584ddc3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:50:52.157852 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18af417-3549-432c-bbfe-e65a584ddc3c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:50:52.161891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ea64cf-2f52-40d0-b007-c9f2fc1dff69 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:50:52.162542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ea64cf-2f52-40d0-b007-c9f2fc1dff69 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:50:52.162820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ea64cf-2f52-40d0-b007-c9f2fc1dff69 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:50:52.163090 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ea64cf-2f52-40d0-b007-c9f2fc1dff69 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:50:52.172054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ea64cf-2f52-40d0-b007-c9f2fc1dff69 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:50:52.173499 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ea64cf-2f52-40d0-b007-c9f2fc1dff69 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:50:52.183395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1075/2165] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:50:52 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:51:02.067235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=44,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:02.178974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:08.293328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdc3777-70c4-4d82-86d7-46f1addea9b5 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:08.372780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdc3777-70c4-4d82-86d7-46f1addea9b5 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:08.380133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47c9624-16e7-4897-b909-7982457506c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:08.380785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47c9624-16e7-4897-b909-7982457506c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:08.381061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47c9624-16e7-4897-b909-7982457506c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:08.381332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47c9624-16e7-4897-b909-7982457506c3 admin admin] PATH_INFO: `/v3/users/51447f0852584a47b59894ee8a130041` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:08.382710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47c9624-16e7-4897-b909-7982457506c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=51447f0852584a47b59894ee8a130041)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:08.388208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47c9624-16e7-4897-b909-7982457506c3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:08.618513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d47c9624-16e7-4897-b909-7982457506c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51447f0852584a47b59894ee8a130041'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:08.619186 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d47c9624-16e7-4897-b909-7982457506c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51447f0852584a47b59894ee8a130041'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:08.626296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d47c9624-16e7-4897-b909-7982457506c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51447f0852584a47b59894ee8a130041'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:08.627668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d47c9624-16e7-4897-b909-7982457506c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51447f0852584a47b59894ee8a130041'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:08.738786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d47c9624-16e7-4897-b909-7982457506c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51447f0852584a47b59894ee8a130041'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:08.771152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1091/2166] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:51:08 2022] DELETE /identity/v3/users/51447f0852584a47b59894ee8a130041 => generated 0 bytes in 481 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:51:08.775806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58992d75-fb0f-4e3a-9d70-2d80e5ddca45 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:08.776263 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58992d75-fb0f-4e3a-9d70-2d80e5ddca45 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:08.776669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58992d75-fb0f-4e3a-9d70-2d80e5ddca45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:08.804166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-58992d75-fb0f-4e3a-9d70-2d80e5ddca45 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:08.804166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-58992d75-fb0f-4e3a-9d70-2d80e5ddca45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:08.807295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84414614-e68b-471d-afb3-bec95ec74706 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:08.807790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84414614-e68b-471d-afb3-bec95ec74706 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:08.808211 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84414614-e68b-471d-afb3-bec95ec74706 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:08.834338 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-84414614-e68b-471d-afb3-bec95ec74706 None None] MFA Rules not processed for user `77b4b56f008c44c081130377cf8ce630`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:08.836254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-84414614-e68b-471d-afb3-bec95ec74706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:08.888894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1076/2167] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:08.901103 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58992d75-fb0f-4e3a-9d70-2d80e5ddca45 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:08.902788 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-58992d75-fb0f-4e3a-9d70-2d80e5ddca45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:08.918382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1092/2168] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:08 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:08.978961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58992d75-fb0f-4e3a-9d70-2d80e5ddca45 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:08.984477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abe396d-faf4-47dc-aa98-9a5bea01f39c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:08.987655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abe396d-faf4-47dc-aa98-9a5bea01f39c service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:08.987952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abe396d-faf4-47dc-aa98-9a5bea01f39c service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:08.988228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abe396d-faf4-47dc-aa98-9a5bea01f39c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:09.017197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abe396d-faf4-47dc-aa98-9a5bea01f39c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:09.017197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abe396d-faf4-47dc-aa98-9a5bea01f39c service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:09.037134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1077/2169] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:51:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:51:10.150654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84414614-e68b-471d-afb3-bec95ec74706 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:10.152238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-84414614-e68b-471d-afb3-bec95ec74706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:10.172421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84414614-e68b-471d-afb3-bec95ec74706 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:10.182946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057367f2-c94a-4f7d-bd88-f83b724cb478 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:10.183747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057367f2-c94a-4f7d-bd88-f83b724cb478 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:10.184125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057367f2-c94a-4f7d-bd88-f83b724cb478 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:10.184462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057367f2-c94a-4f7d-bd88-f83b724cb478 admin admin] PATH_INFO: `/v3/projects/daac215191a944abb73c2e297241f09e` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:10.186189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057367f2-c94a-4f7d-bd88-f83b724cb478 admin admin] RBAC: Authorizing `identity:delete_project(project_id=daac215191a944abb73c2e297241f09e)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:10.189668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057367f2-c94a-4f7d-bd88-f83b724cb478 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:10.245534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:10.382296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-057367f2-c94a-4f7d-bd88-f83b724cb478 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'daac215191a944abb73c2e297241f09e', 'request_id': 'req-057367f2-c94a-4f7d-bd88-f83b724cb478'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:10.387515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-057367f2-c94a-4f7d-bd88-f83b724cb478 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'daac215191a944abb73c2e297241f09e', 'request_id': 'req-057367f2-c94a-4f7d-bd88-f83b724cb478'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:10.567445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-057367f2-c94a-4f7d-bd88-f83b724cb478 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'daac215191a944abb73c2e297241f09e', 'request_id': 'req-057367f2-c94a-4f7d-bd88-f83b724cb478'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:10.587691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-057367f2-c94a-4f7d-bd88-f83b724cb478 admin admin] The token cache is being invalidate because project daac215191a944abb73c2e297241f09e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:51:10.587691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-057367f2-c94a-4f7d-bd88-f83b724cb478 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:10.587691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-057367f2-c94a-4f7d-bd88-f83b724cb478 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:10.587691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1093/2170] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:51:10 2022] DELETE /identity/v3/projects/daac215191a944abb73c2e297241f09e => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:51:10.589737 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82c1d44-8b5e-4280-a933-713c5bc2b2a6 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:10.590262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82c1d44-8b5e-4280-a933-713c5bc2b2a6 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:10.590262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82c1d44-8b5e-4280-a933-713c5bc2b2a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:10.635598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-b82c1d44-8b5e-4280-a933-713c5bc2b2a6 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:10.635598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b82c1d44-8b5e-4280-a933-713c5bc2b2a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:10.726114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1078/2171] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:10.726663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff73a3c-0758-4112-beb8-52b6eab10c7c None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:10.726957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff73a3c-0758-4112-beb8-52b6eab10c7c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:10.727234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff73a3c-0758-4112-beb8-52b6eab10c7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:10.751825 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-4ff73a3c-0758-4112-beb8-52b6eab10c7c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:10.753097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4ff73a3c-0758-4112-beb8-52b6eab10c7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:10.765903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1094/2172] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:51:10.773050 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82c1d44-8b5e-4280-a933-713c5bc2b2a6 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:10.784858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82c1d44-8b5e-4280-a933-713c5bc2b2a6 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:10.790426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3175c4ea-7805-4d42-a954-4fca8f39f501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:10.790917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3175c4ea-7805-4d42-a954-4fca8f39f501 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:10.791197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3175c4ea-7805-4d42-a954-4fca8f39f501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:10.791456 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3175c4ea-7805-4d42-a954-4fca8f39f501 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:10.792339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3175c4ea-7805-4d42-a954-4fca8f39f501 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:51:10.792814 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3175c4ea-7805-4d42-a954-4fca8f39f501 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:10.818222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3175c4ea-7805-4d42-a954-4fca8f39f501 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:10.818222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1079/2173] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:51:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:51:10.821336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff73a3c-0758-4112-beb8-52b6eab10c7c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:10.832643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff73a3c-0758-4112-beb8-52b6eab10c7c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:10.837711 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cbb7b8-4f67-4429-972e-3a3743ff7d20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:10.838450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cbb7b8-4f67-4429-972e-3a3743ff7d20 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:10.838928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cbb7b8-4f67-4429-972e-3a3743ff7d20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:10.839307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cbb7b8-4f67-4429-972e-3a3743ff7d20 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:10.840040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cbb7b8-4f67-4429-972e-3a3743ff7d20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:10.842337 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cbb7b8-4f67-4429-972e-3a3743ff7d20 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:11.042166 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1095/2174] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:51:10 2022] POST /identity/v3/projects => generated 397 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:51:11.046571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3175c4ea-7805-4d42-a954-4fca8f39f501 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:11.057758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3175c4ea-7805-4d42-a954-4fca8f39f501 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:11.087288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:11.117271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013aa3d4-e8b5-439f-a9ef-cbc5ad41d013 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:11.117986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013aa3d4-e8b5-439f-a9ef-cbc5ad41d013 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:11.118460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013aa3d4-e8b5-439f-a9ef-cbc5ad41d013 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:11.118756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013aa3d4-e8b5-439f-a9ef-cbc5ad41d013 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:11.119476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013aa3d4-e8b5-439f-a9ef-cbc5ad41d013 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:11.121597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013aa3d4-e8b5-439f-a9ef-cbc5ad41d013 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:11.230032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1080/2175] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:51:11 2022] POST /identity/v3/users => generated 299 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:51:11.234236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cbb7b8-4f67-4429-972e-3a3743ff7d20 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:11.246725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cbb7b8-4f67-4429-972e-3a3743ff7d20 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:11.252733 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574144a2-4d13-498d-8ac2-d02eceeae7ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:11.253444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574144a2-4d13-498d-8ac2-d02eceeae7ef admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:11.253794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574144a2-4d13-498d-8ac2-d02eceeae7ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:11.254138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574144a2-4d13-498d-8ac2-d02eceeae7ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:11.254893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574144a2-4d13-498d-8ac2-d02eceeae7ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:11.256949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574144a2-4d13-498d-8ac2-d02eceeae7ef admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:11.274089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1096/2176] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:51:11 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:51:11.276469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013aa3d4-e8b5-439f-a9ef-cbc5ad41d013 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:11.288171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013aa3d4-e8b5-439f-a9ef-cbc5ad41d013 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:11.293356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd34fdd0-0041-436c-b120-02327ffc6721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:11.294226 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd34fdd0-0041-436c-b120-02327ffc6721 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:11.294602 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd34fdd0-0041-436c-b120-02327ffc6721 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:11.294941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd34fdd0-0041-436c-b120-02327ffc6721 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:11.295685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd34fdd0-0041-436c-b120-02327ffc6721 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:11.297356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd34fdd0-0041-436c-b120-02327ffc6721 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:11.309840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1081/2177] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:51:11 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:51:11.318475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574144a2-4d13-498d-8ac2-d02eceeae7ef admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:11.329468 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574144a2-4d13-498d-8ac2-d02eceeae7ef admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:11.346332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad47c6f6-8b45-47ef-8b0f-7a5126636aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:11.346332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad47c6f6-8b45-47ef-8b0f-7a5126636aea admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:11.346332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad47c6f6-8b45-47ef-8b0f-7a5126636aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:11.346332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad47c6f6-8b45-47ef-8b0f-7a5126636aea admin admin] PATH_INFO: `/v3/projects/70e2906638c14441a09c5b07e5104e4d/users/ce9f1d42c6df4784b24600ab142bda4f/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:11.366020 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad47c6f6-8b45-47ef-8b0f-7a5126636aea admin admin] RBAC: Authorizing `identity:create_grant(project_id=70e2906638c14441a09c5b07e5104e4d, user_id=ce9f1d42c6df4784b24600ab142bda4f, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:11.368736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad47c6f6-8b45-47ef-8b0f-7a5126636aea admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:11.456286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1097/2178] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:51:11 2022] PUT /identity/v3/projects/70e2906638c14441a09c5b07e5104e4d/users/ce9f1d42c6df4784b24600ab142bda4f/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:51:11.467628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb9c185-4c4f-44fd-8ebd-348092d76535 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:11.467947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb9c185-4c4f-44fd-8ebd-348092d76535 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:11.468224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb9c185-4c4f-44fd-8ebd-348092d76535 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:11.499796 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-feb9c185-4c4f-44fd-8ebd-348092d76535 None None] MFA Rules not processed for user `ce9f1d42c6df4784b24600ab142bda4f`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:11.501041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-feb9c185-4c4f-44fd-8ebd-348092d76535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:11.589961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1082/2179] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:11 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:11.603657 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad47c6f6-8b45-47ef-8b0f-7a5126636aea admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:11.606285 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ad47c6f6-8b45-47ef-8b0f-7a5126636aea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:11.670884 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad47c6f6-8b45-47ef-8b0f-7a5126636aea admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:11.674764 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482d27d1-b8c4-4c3e-a3e4-d0e737920317 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:11.675370 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482d27d1-b8c4-4c3e-a3e4-d0e737920317 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:11.675647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482d27d1-b8c4-4c3e-a3e4-d0e737920317 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:11.675917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482d27d1-b8c4-4c3e-a3e4-d0e737920317 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:11.684229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482d27d1-b8c4-4c3e-a3e4-d0e737920317 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:11.685683 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482d27d1-b8c4-4c3e-a3e4-d0e737920317 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:11.695443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1098/2180] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:51:11 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:51:14.974405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05be6741-9161-4938-bbd4-284f10818797 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:14.974981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05be6741-9161-4938-bbd4-284f10818797 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:14.974981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05be6741-9161-4938-bbd4-284f10818797 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:14.995738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-05be6741-9161-4938-bbd4-284f10818797 None None] MFA Rules not processed for user `0772f8b4306f4e239f075f7b1c13790f`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:14.996959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-05be6741-9161-4938-bbd4-284f10818797 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:15.059219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1083/2181] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:14 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:17.250266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482d27d1-b8c4-4c3e-a3e4-d0e737920317 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:17.251997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-482d27d1-b8c4-4c3e-a3e4-d0e737920317 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:17.308149 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:17.329120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482d27d1-b8c4-4c3e-a3e4-d0e737920317 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:17.336394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849a97cd-7139-4fbd-bce0-3921ae5ab182 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:17.337137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849a97cd-7139-4fbd-bce0-3921ae5ab182 service placement] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:17.337423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849a97cd-7139-4fbd-bce0-3921ae5ab182 service placement] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:17.337697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849a97cd-7139-4fbd-bce0-3921ae5ab182 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:17.339525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-849a97cd-7139-4fbd-bce0-3921ae5ab182 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:17.353725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849a97cd-7139-4fbd-bce0-3921ae5ab182 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:17.355468 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849a97cd-7139-4fbd-bce0-3921ae5ab182 service placement] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:17.366708 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1099/2182] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:51:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:51:25.050751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=77,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:25.476255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa82263-f72f-4b31-8afa-1f1dc226c6f1 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:25.476537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa82263-f72f-4b31-8afa-1f1dc226c6f1 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:25.476939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa82263-f72f-4b31-8afa-1f1dc226c6f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:25.513278 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-9aa82263-f72f-4b31-8afa-1f1dc226c6f1 None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:25.514563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-9aa82263-f72f-4b31-8afa-1f1dc226c6f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:25.626186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1084/2183] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:51:25 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:25.673367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849a97cd-7139-4fbd-bce0-3921ae5ab182 service placement] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:25.673367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-849a97cd-7139-4fbd-bce0-3921ae5ab182 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:25.756229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849a97cd-7139-4fbd-bce0-3921ae5ab182 service placement] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:25.756229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ee501e-4c70-42db-a6ba-474d7ca1738e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:25.756229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ee501e-4c70-42db-a6ba-474d7ca1738e service swift] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:25.756229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ee501e-4c70-42db-a6ba-474d7ca1738e service swift] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:25.756535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ee501e-4c70-42db-a6ba-474d7ca1738e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:25.764362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ee501e-4c70-42db-a6ba-474d7ca1738e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:25.766041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ee501e-4c70-42db-a6ba-474d7ca1738e service swift] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:25.776034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1100/2184] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:51:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:51:28.648923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa82263-f72f-4b31-8afa-1f1dc226c6f1 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:28.832468 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa82263-f72f-4b31-8afa-1f1dc226c6f1 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:28.838617 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:28.839722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:28.840227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:28.840565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] PATH_INFO: `/v3/users/ce9f1d42c6df4784b24600ab142bda4f` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:28.841806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce9f1d42c6df4784b24600ab142bda4f)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:28.844114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:29.128835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce9f1d42c6df4784b24600ab142bda4f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:29.129534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce9f1d42c6df4784b24600ab142bda4f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:29.150417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce9f1d42c6df4784b24600ab142bda4f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:29.177956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce9f1d42c6df4784b24600ab142bda4f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:29.273941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce9f1d42c6df4784b24600ab142bda4f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:29.286806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1085/2185] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:51:28 2022] DELETE /identity/v3/users/ce9f1d42c6df4784b24600ab142bda4f => generated 0 bytes in 635 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:51:29.293333 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7484d69d-f20c-44d3-b76e-d3e1129fa7fa None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:29.293333 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7484d69d-f20c-44d3-b76e-d3e1129fa7fa None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:29.293333 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7484d69d-f20c-44d3-b76e-d3e1129fa7fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:29.316272 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-7484d69d-f20c-44d3-b76e-d3e1129fa7fa None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:29.318697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7484d69d-f20c-44d3-b76e-d3e1129fa7fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:29.392202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1101/2186] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:51:29.424474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:29.427152 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:29.500498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:29.561842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950c3aaa-0fe4-4c38-aa90-42bc92837225 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:29.567660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaeaf4b-eda6-44b4-a3ef-517188a1453e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:29.568325 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaeaf4b-eda6-44b4-a3ef-517188a1453e service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:29.568607 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaeaf4b-eda6-44b4-a3ef-517188a1453e service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:29.569341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaeaf4b-eda6-44b4-a3ef-517188a1453e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:29.582692 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faaeaf4b-eda6-44b4-a3ef-517188a1453e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:29.584110 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faaeaf4b-eda6-44b4-a3ef-517188a1453e service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:29.597906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1086/2187] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:51:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:51:30.859581 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7484d69d-f20c-44d3-b76e-d3e1129fa7fa None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:30.861325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7484d69d-f20c-44d3-b76e-d3e1129fa7fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:30.875723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7484d69d-f20c-44d3-b76e-d3e1129fa7fa None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:30.883293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cce6b20-5f84-49bf-80cd-f84ab6875781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:30.884205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cce6b20-5f84-49bf-80cd-f84ab6875781 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:30.884699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cce6b20-5f84-49bf-80cd-f84ab6875781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:30.885147 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cce6b20-5f84-49bf-80cd-f84ab6875781 admin admin] PATH_INFO: `/v3/projects/70e2906638c14441a09c5b07e5104e4d` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:30.886811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cce6b20-5f84-49bf-80cd-f84ab6875781 admin admin] RBAC: Authorizing `identity:delete_project(project_id=70e2906638c14441a09c5b07e5104e4d)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:30.888845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cce6b20-5f84-49bf-80cd-f84ab6875781 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:31.064242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2cce6b20-5f84-49bf-80cd-f84ab6875781 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '70e2906638c14441a09c5b07e5104e4d', 'request_id': 'req-2cce6b20-5f84-49bf-80cd-f84ab6875781'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:31.074993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2cce6b20-5f84-49bf-80cd-f84ab6875781 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '70e2906638c14441a09c5b07e5104e4d', 'request_id': 'req-2cce6b20-5f84-49bf-80cd-f84ab6875781'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:31.081927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:31.192641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2cce6b20-5f84-49bf-80cd-f84ab6875781 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '70e2906638c14441a09c5b07e5104e4d', 'request_id': 'req-2cce6b20-5f84-49bf-80cd-f84ab6875781'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:31.216298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2cce6b20-5f84-49bf-80cd-f84ab6875781 admin admin] The token cache is being invalidate because project 70e2906638c14441a09c5b07e5104e4d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:51:31.216576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2cce6b20-5f84-49bf-80cd-f84ab6875781 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:31.218638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2cce6b20-5f84-49bf-80cd-f84ab6875781 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:31.226927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1102/2188] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:51:30 2022] DELETE /identity/v3/projects/70e2906638c14441a09c5b07e5104e4d => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:51:31.238989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223902df-c4b6-45b2-bb36-bba693a0d6f9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:31.238989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223902df-c4b6-45b2-bb36-bba693a0d6f9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:31.238989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223902df-c4b6-45b2-bb36-bba693a0d6f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:31.317276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-223902df-c4b6-45b2-bb36-bba693a0d6f9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:31.320244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-223902df-c4b6-45b2-bb36-bba693a0d6f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:31.434233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1087/2189] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:31.447207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d9698-5c6e-4737-a997-57f2b4a40593 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:31.451504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d9698-5c6e-4737-a997-57f2b4a40593 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:31.451504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d9698-5c6e-4737-a997-57f2b4a40593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:31.480921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-739d9698-5c6e-4737-a997-57f2b4a40593 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:31.482765 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-739d9698-5c6e-4737-a997-57f2b4a40593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:31.500715 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1103/2190] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:31.506519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223902df-c4b6-45b2-bb36-bba693a0d6f9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:31.523258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223902df-c4b6-45b2-bb36-bba693a0d6f9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:31.530374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313224a6-4285-445a-822c-48fe0c379465 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:31.531254 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313224a6-4285-445a-822c-48fe0c379465 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:31.531707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313224a6-4285-445a-822c-48fe0c379465 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:31.532191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313224a6-4285-445a-822c-48fe0c379465 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:31.534436 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313224a6-4285-445a-822c-48fe0c379465 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:51:31.534436 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313224a6-4285-445a-822c-48fe0c379465 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:31.536459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313224a6-4285-445a-822c-48fe0c379465 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:31.567536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1088/2191] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:51:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:51:31.573054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739d9698-5c6e-4737-a997-57f2b4a40593 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:31.584568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739d9698-5c6e-4737-a997-57f2b4a40593 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:31.590856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6716eea5-d3cf-4c3e-9e8d-b2ab03d1cf96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:31.591693 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6716eea5-d3cf-4c3e-9e8d-b2ab03d1cf96 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:31.597571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6716eea5-d3cf-4c3e-9e8d-b2ab03d1cf96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:31.597571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6716eea5-d3cf-4c3e-9e8d-b2ab03d1cf96 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:31.597571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6716eea5-d3cf-4c3e-9e8d-b2ab03d1cf96 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:31.597571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6716eea5-d3cf-4c3e-9e8d-b2ab03d1cf96 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:31.720870 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1104/2192] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:51:31 2022] POST /identity/v3/projects => generated 377 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:51:31.726460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313224a6-4285-445a-822c-48fe0c379465 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:31.740460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313224a6-4285-445a-822c-48fe0c379465 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:31.817933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5248b834-818e-40fd-88b6-ab56ee35956e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:31.818587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5248b834-818e-40fd-88b6-ab56ee35956e admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:31.818849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5248b834-818e-40fd-88b6-ab56ee35956e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:31.819116 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5248b834-818e-40fd-88b6-ab56ee35956e admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:31.819735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5248b834-818e-40fd-88b6-ab56ee35956e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:31.822905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5248b834-818e-40fd-88b6-ab56ee35956e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:31.932942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1089/2193] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:51:31 2022] POST /identity/v3/users => generated 289 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:51:31.939063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6716eea5-d3cf-4c3e-9e8d-b2ab03d1cf96 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:31.956033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6716eea5-d3cf-4c3e-9e8d-b2ab03d1cf96 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:31.971691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70285cc2-8d5f-46ac-99b1-aaef46a8a18d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:31.972335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70285cc2-8d5f-46ac-99b1-aaef46a8a18d admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:31.972781 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70285cc2-8d5f-46ac-99b1-aaef46a8a18d admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:31.973629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70285cc2-8d5f-46ac-99b1-aaef46a8a18d admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:31.974091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70285cc2-8d5f-46ac-99b1-aaef46a8a18d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:31.975727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70285cc2-8d5f-46ac-99b1-aaef46a8a18d admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:31.987558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1105/2194] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:51:31 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:51:31.992630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5248b834-818e-40fd-88b6-ab56ee35956e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:32.007419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5248b834-818e-40fd-88b6-ab56ee35956e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:32.013859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08fb0cf-be4b-4586-a2c3-36968e0baa69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:32.014731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08fb0cf-be4b-4586-a2c3-36968e0baa69 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:32.015171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08fb0cf-be4b-4586-a2c3-36968e0baa69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:32.015582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08fb0cf-be4b-4586-a2c3-36968e0baa69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:32.017272 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08fb0cf-be4b-4586-a2c3-36968e0baa69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:32.019734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08fb0cf-be4b-4586-a2c3-36968e0baa69 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:32.042186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1090/2195] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:51:31 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:51:32.059770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70285cc2-8d5f-46ac-99b1-aaef46a8a18d admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:32.071119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70285cc2-8d5f-46ac-99b1-aaef46a8a18d admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:32.082201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75395e65-8c45-4cfb-b936-7036b0a607b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:32.088505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75395e65-8c45-4cfb-b936-7036b0a607b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:32.088505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75395e65-8c45-4cfb-b936-7036b0a607b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:32.088505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75395e65-8c45-4cfb-b936-7036b0a607b6 admin admin] PATH_INFO: `/v3/projects/33235ed2b4cd48e7ac205143b814c6fb/users/71ff73cade8b42c7ae57be1e3ef0e028/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:32.108908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75395e65-8c45-4cfb-b936-7036b0a607b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33235ed2b4cd48e7ac205143b814c6fb, user_id=71ff73cade8b42c7ae57be1e3ef0e028, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:32.114531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75395e65-8c45-4cfb-b936-7036b0a607b6 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:32.275661 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1106/2196] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:51:32 2022] PUT /identity/v3/projects/33235ed2b4cd48e7ac205143b814c6fb/users/71ff73cade8b42c7ae57be1e3ef0e028/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:51:32.280846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5675e60a-9e81-4912-af50-7f426b9c9064 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:32.281298 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5675e60a-9e81-4912-af50-7f426b9c9064 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:32.281404 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5675e60a-9e81-4912-af50-7f426b9c9064 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:32.308818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-5675e60a-9e81-4912-af50-7f426b9c9064 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:32.310560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5675e60a-9e81-4912-af50-7f426b9c9064 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:32.354405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:32.395183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1091/2197] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:32.406126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75395e65-8c45-4cfb-b936-7036b0a607b6 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:32.409077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-75395e65-8c45-4cfb-b936-7036b0a607b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:32.505540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75395e65-8c45-4cfb-b936-7036b0a607b6 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:32.510249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c653c669-abc1-4ddd-b2b0-21afbca06924 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:32.510882 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c653c669-abc1-4ddd-b2b0-21afbca06924 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:32.511159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c653c669-abc1-4ddd-b2b0-21afbca06924 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:32.511454 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c653c669-abc1-4ddd-b2b0-21afbca06924 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:32.521943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c653c669-abc1-4ddd-b2b0-21afbca06924 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:32.523607 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c653c669-abc1-4ddd-b2b0-21afbca06924 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:32.537440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1107/2198] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:51:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:51:33.284144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5675e60a-9e81-4912-af50-7f426b9c9064 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:33.285682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5675e60a-9e81-4912-af50-7f426b9c9064 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:33.306261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5675e60a-9e81-4912-af50-7f426b9c9064 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:33.321710 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:33.322972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:33.322972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:33.322972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] PATH_INFO: `/v3/users/40c5fca261e24b49adca1b04fdcba5c6` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:33.327614 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] RBAC: Authorizing `identity:delete_user(user_id=40c5fca261e24b49adca1b04fdcba5c6)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:33.331047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:33.541200 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40c5fca261e24b49adca1b04fdcba5c6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:33.542189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40c5fca261e24b49adca1b04fdcba5c6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:33.567276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40c5fca261e24b49adca1b04fdcba5c6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:33.602787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40c5fca261e24b49adca1b04fdcba5c6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:33.719048 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40c5fca261e24b49adca1b04fdcba5c6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:33.729200 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1092/2199] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:51:33 2022] DELETE /identity/v3/users/40c5fca261e24b49adca1b04fdcba5c6 => generated 0 bytes in 446 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:51:33.733482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ef5112-aac6-4aa1-b968-6689c2767b57 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:33.733992 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ef5112-aac6-4aa1-b968-6689c2767b57 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:33.734071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ef5112-aac6-4aa1-b968-6689c2767b57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:33.775223 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d0ef5112-aac6-4aa1-b968-6689c2767b57 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:33.776464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d0ef5112-aac6-4aa1-b968-6689c2767b57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:33.888379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1108/2200] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:33.899454 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:33.901332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:34.012423 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69136f70-46a4-4df9-9120-c236037a490c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:34.032708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e5c8f9-118f-4875-8ee4-419a7d8643f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:34.038276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e5c8f9-118f-4875-8ee4-419a7d8643f2 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:34.038276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e5c8f9-118f-4875-8ee4-419a7d8643f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:34.038276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e5c8f9-118f-4875-8ee4-419a7d8643f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:34.050989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e5c8f9-118f-4875-8ee4-419a7d8643f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:34.052493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e5c8f9-118f-4875-8ee4-419a7d8643f2 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:34.073422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1093/2201] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:51:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:51:35.091350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ef5112-aac6-4aa1-b968-6689c2767b57 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:35.094221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d0ef5112-aac6-4aa1-b968-6689c2767b57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:35.127518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ef5112-aac6-4aa1-b968-6689c2767b57 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:35.143229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:35.143903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:35.144180 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:35.144444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784 admin admin] PATH_INFO: `/v3/projects/6552bf143d28461c98362adc695d06a3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:35.147462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6552bf143d28461c98362adc695d06a3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:35.150518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:35.205234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:DELETE=2,keystone:SELECT=96,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:35.315946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6552bf143d28461c98362adc695d06a3', 'request_id': 'req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:35.326198 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6552bf143d28461c98362adc695d06a3', 'request_id': 'req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:35.435085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6552bf143d28461c98362adc695d06a3', 'request_id': 'req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:35.460425 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784 admin admin] The token cache is being invalidate because project 6552bf143d28461c98362adc695d06a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:51:35.461187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:35.462869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c9bfd6bf-3eaf-4258-9dc2-2d3a80cad784 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:35.469850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1109/2202] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:51:35 2022] DELETE /identity/v3/projects/6552bf143d28461c98362adc695d06a3 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:51:35.475820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc19975-551e-42a3-a2bd-fe44ccbc3ca7 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:35.476777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc19975-551e-42a3-a2bd-fe44ccbc3ca7 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:35.476777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc19975-551e-42a3-a2bd-fe44ccbc3ca7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:35.535259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-cdc19975-551e-42a3-a2bd-fe44ccbc3ca7 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:35.536806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-cdc19975-551e-42a3-a2bd-fe44ccbc3ca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:35.660664 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:35.708533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1094/2203] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:51:35.720551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1ebdaf-c6da-4531-a1ac-2faf043ab9a9 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:35.720821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1ebdaf-c6da-4531-a1ac-2faf043ab9a9 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:35.721098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1ebdaf-c6da-4531-a1ac-2faf043ab9a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:35.748135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5f1ebdaf-c6da-4531-a1ac-2faf043ab9a9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:35.749705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5f1ebdaf-c6da-4531-a1ac-2faf043ab9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:35.775783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1110/2204] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:51:35.779683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc19975-551e-42a3-a2bd-fe44ccbc3ca7 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:35.793300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc19975-551e-42a3-a2bd-fe44ccbc3ca7 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:35.808961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d635ff5-f346-41f0-b5a2-d228f51c2c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:35.812625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d635ff5-f346-41f0-b5a2-d228f51c2c15 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:35.814676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d635ff5-f346-41f0-b5a2-d228f51c2c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:35.815253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d635ff5-f346-41f0-b5a2-d228f51c2c15 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:35.818779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d635ff5-f346-41f0-b5a2-d228f51c2c15 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:51:35.822213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d635ff5-f346-41f0-b5a2-d228f51c2c15 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:35.826430 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d635ff5-f346-41f0-b5a2-d228f51c2c15 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:35.858101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1095/2205] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:51:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:51:35.861220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1ebdaf-c6da-4531-a1ac-2faf043ab9a9 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:35.881383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1ebdaf-c6da-4531-a1ac-2faf043ab9a9 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:35.896242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ec534f-e944-4a42-8391-067dba0d2e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:35.897210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ec534f-e944-4a42-8391-067dba0d2e75 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:35.897370 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ec534f-e944-4a42-8391-067dba0d2e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:35.897591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ec534f-e944-4a42-8391-067dba0d2e75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:35.898256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ec534f-e944-4a42-8391-067dba0d2e75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:35.900320 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ec534f-e944-4a42-8391-067dba0d2e75 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:36.102792 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1111/2206] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:51:35 2022] POST /identity/v3/projects => generated 385 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:51:36.108742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d635ff5-f346-41f0-b5a2-d228f51c2c15 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:36.132664 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d635ff5-f346-41f0-b5a2-d228f51c2c15 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:36.294392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eee2f9-3015-4206-89ed-5336d7db2591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:36.295061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eee2f9-3015-4206-89ed-5336d7db2591 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:36.295343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eee2f9-3015-4206-89ed-5336d7db2591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:36.295605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eee2f9-3015-4206-89ed-5336d7db2591 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:36.296283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6eee2f9-3015-4206-89ed-5336d7db2591 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:36.298375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6eee2f9-3015-4206-89ed-5336d7db2591 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:36.446927 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1096/2207] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:51:36 2022] POST /identity/v3/users => generated 293 bytes in 338 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:51:36.450976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ec534f-e944-4a42-8391-067dba0d2e75 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:36.463189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ec534f-e944-4a42-8391-067dba0d2e75 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:36.468224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfcbb44-c55e-48bd-b99d-24a8ba37c719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:36.469102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfcbb44-c55e-48bd-b99d-24a8ba37c719 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:36.469986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfcbb44-c55e-48bd-b99d-24a8ba37c719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:36.472102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfcbb44-c55e-48bd-b99d-24a8ba37c719 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:36.472102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfcbb44-c55e-48bd-b99d-24a8ba37c719 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:36.473133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfcbb44-c55e-48bd-b99d-24a8ba37c719 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:36.506721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1112/2208] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:51:36 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:51:36.508304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eee2f9-3015-4206-89ed-5336d7db2591 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:36.545705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6eee2f9-3015-4206-89ed-5336d7db2591 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:36.574992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ae3cf5-54cd-4c57-b38a-2d6ce0a3f8e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:36.574992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ae3cf5-54cd-4c57-b38a-2d6ce0a3f8e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:36.574992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ae3cf5-54cd-4c57-b38a-2d6ce0a3f8e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:36.577926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ae3cf5-54cd-4c57-b38a-2d6ce0a3f8e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:36.578798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ae3cf5-54cd-4c57-b38a-2d6ce0a3f8e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:36.580782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ae3cf5-54cd-4c57-b38a-2d6ce0a3f8e5 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:36.624123 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfcbb44-c55e-48bd-b99d-24a8ba37c719 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:36.638780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1097/2209] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:51:36 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:51:36.649428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfcbb44-c55e-48bd-b99d-24a8ba37c719 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:36.667265 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64df6e3-2396-4f6c-a741-fe59f38e3f60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:36.669792 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64df6e3-2396-4f6c-a741-fe59f38e3f60 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:36.670265 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64df6e3-2396-4f6c-a741-fe59f38e3f60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:36.670844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64df6e3-2396-4f6c-a741-fe59f38e3f60 admin admin] PATH_INFO: `/v3/projects/d97ce3d33a254ff8b04aecd1c9e0b5f6/users/63b8afec8f7b4d59af334faec7992bb8/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:36.718140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64df6e3-2396-4f6c-a741-fe59f38e3f60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d97ce3d33a254ff8b04aecd1c9e0b5f6, user_id=63b8afec8f7b4d59af334faec7992bb8, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:36.722565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64df6e3-2396-4f6c-a741-fe59f38e3f60 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:36.856870 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1113/2210] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:51:36 2022] PUT /identity/v3/projects/d97ce3d33a254ff8b04aecd1c9e0b5f6/users/63b8afec8f7b4d59af334faec7992bb8/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:51:36.858813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfb4b99-2fa6-413d-bb3f-ae050edd3954 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:36.860391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfb4b99-2fa6-413d-bb3f-ae050edd3954 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:36.861415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfb4b99-2fa6-413d-bb3f-ae050edd3954 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:36.947692 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-9dfb4b99-2fa6-413d-bb3f-ae050edd3954 None None] MFA Rules not processed for user `63b8afec8f7b4d59af334faec7992bb8`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:36.948965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-9dfb4b99-2fa6-413d-bb3f-ae050edd3954 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:37.041397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1098/2211] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:36 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:37.054350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64df6e3-2396-4f6c-a741-fe59f38e3f60 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:37.056806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f64df6e3-2396-4f6c-a741-fe59f38e3f60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:37.226463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64df6e3-2396-4f6c-a741-fe59f38e3f60 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:37.260049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128f7e21-a2a0-4c86-93fb-757dfc5be31d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:37.262102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128f7e21-a2a0-4c86-93fb-757dfc5be31d service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:37.270448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128f7e21-a2a0-4c86-93fb-757dfc5be31d service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:37.270448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128f7e21-a2a0-4c86-93fb-757dfc5be31d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:37.304814 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128f7e21-a2a0-4c86-93fb-757dfc5be31d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:37.309054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128f7e21-a2a0-4c86-93fb-757dfc5be31d service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:37.357906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1114/2212] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:51:37 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:51:42.443776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f275100-fdcd-48c5-9720-bee01e3b12f7 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:42.444389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f275100-fdcd-48c5-9720-bee01e3b12f7 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:42.444528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f275100-fdcd-48c5-9720-bee01e3b12f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:42.497845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-5f275100-fdcd-48c5-9720-bee01e3b12f7 None None] MFA Rules not processed for user `ae04cdfc35964cf58502cefd1cc912e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:42.508036 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5f275100-fdcd-48c5-9720-bee01e3b12f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:42.612472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:42.640086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1099/2213] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:42 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:42.910709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128f7e21-a2a0-4c86-93fb-757dfc5be31d service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:43.025962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=91,keystone:DELETE=6,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:43.047542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128f7e21-a2a0-4c86-93fb-757dfc5be31d service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:43.053427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:43.054282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:43.054673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:43.055343 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] PATH_INFO: `/v3/users/63b8afec8f7b4d59af334faec7992bb8` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:43.056609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] RBAC: Authorizing `identity:delete_user(user_id=63b8afec8f7b4d59af334faec7992bb8)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:43.058858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:43.517817 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '63b8afec8f7b4d59af334faec7992bb8'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:43.518530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '63b8afec8f7b4d59af334faec7992bb8'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:43.525520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '63b8afec8f7b4d59af334faec7992bb8'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:43.526227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '63b8afec8f7b4d59af334faec7992bb8'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:43.664576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '63b8afec8f7b4d59af334faec7992bb8'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:43.685135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1115/2214] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:51:42 2022] DELETE /identity/v3/users/63b8afec8f7b4d59af334faec7992bb8 => generated 0 bytes in 775 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:51:43.689795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ac4cf8-31b4-48b7-9691-da50a7cf150c None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:43.690351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ac4cf8-31b4-48b7-9691-da50a7cf150c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:43.690351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ac4cf8-31b4-48b7-9691-da50a7cf150c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:43.710746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-86ac4cf8-31b4-48b7-9691-da50a7cf150c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:43.711767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-86ac4cf8-31b4-48b7-9691-da50a7cf150c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:43.803011 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1100/2215] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:43.813296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:43.814780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:43.880390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc6096e-2308-4851-bb72-67c2956c42ec admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:43.884071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7191e3a-96bc-430d-ab22-d7090f423789 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:43.884808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7191e3a-96bc-430d-ab22-d7090f423789 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:43.885098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7191e3a-96bc-430d-ab22-d7090f423789 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:43.885376 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7191e3a-96bc-430d-ab22-d7090f423789 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:43.897262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7191e3a-96bc-430d-ab22-d7090f423789 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:43.902924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7191e3a-96bc-430d-ab22-d7090f423789 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:43.917271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1116/2216] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:51:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:51:44.424151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567307a7-df05-453b-8691-feddf7b5aaa7 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:44.424475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567307a7-df05-453b-8691-feddf7b5aaa7 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:44.424701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567307a7-df05-453b-8691-feddf7b5aaa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:44.545292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-567307a7-df05-453b-8691-feddf7b5aaa7 None None] MFA Rules not processed for user `71ff73cade8b42c7ae57be1e3ef0e028`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:44.546619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-567307a7-df05-453b-8691-feddf7b5aaa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:44.724983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1101/2217] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:44 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:44.726035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7191e3a-96bc-430d-ab22-d7090f423789 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:44.727670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b7191e3a-96bc-430d-ab22-d7090f423789 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:44.745670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7191e3a-96bc-430d-ab22-d7090f423789 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:44.763490 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ba9a09-5291-4fd7-846b-503083483e78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:44.765305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ba9a09-5291-4fd7-846b-503083483e78 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:44.765305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ba9a09-5291-4fd7-846b-503083483e78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:44.765305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ba9a09-5291-4fd7-846b-503083483e78 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:44.766291 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ba9a09-5291-4fd7-846b-503083483e78 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:44.769355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ba9a09-5291-4fd7-846b-503083483e78 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:44.798496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567307a7-df05-453b-8691-feddf7b5aaa7 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:44.799930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-567307a7-df05-453b-8691-feddf7b5aaa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:44.825431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567307a7-df05-453b-8691-feddf7b5aaa7 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:44.843088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448fe05b-645e-4e0c-98d1-281c89f20293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:44.845134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448fe05b-645e-4e0c-98d1-281c89f20293 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:44.845577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448fe05b-645e-4e0c-98d1-281c89f20293 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:44.845849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448fe05b-645e-4e0c-98d1-281c89f20293 admin admin] PATH_INFO: `/v3/projects/d97ce3d33a254ff8b04aecd1c9e0b5f6` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:44.847336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448fe05b-645e-4e0c-98d1-281c89f20293 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d97ce3d33a254ff8b04aecd1c9e0b5f6)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:44.849734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448fe05b-645e-4e0c-98d1-281c89f20293 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:44.902753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1117/2218] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:51:44 2022] POST /identity/v3/projects => generated 377 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:51:44.905091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ba9a09-5291-4fd7-846b-503083483e78 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:45.017723 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:45.030280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ba9a09-5291-4fd7-846b-503083483e78 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:45.033671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-448fe05b-645e-4e0c-98d1-281c89f20293 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd97ce3d33a254ff8b04aecd1c9e0b5f6', 'request_id': 'req-448fe05b-645e-4e0c-98d1-281c89f20293'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:45.052307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-448fe05b-645e-4e0c-98d1-281c89f20293 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd97ce3d33a254ff8b04aecd1c9e0b5f6', 'request_id': 'req-448fe05b-645e-4e0c-98d1-281c89f20293'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:45.061845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-448fe05b-645e-4e0c-98d1-281c89f20293 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd97ce3d33a254ff8b04aecd1c9e0b5f6', 'request_id': 'req-448fe05b-645e-4e0c-98d1-281c89f20293'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:45.096171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9921944-ddeb-4f19-903d-2f810b1a77d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:45.097094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9921944-ddeb-4f19-903d-2f810b1a77d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.097583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9921944-ddeb-4f19-903d-2f810b1a77d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.098059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9921944-ddeb-4f19-903d-2f810b1a77d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:45.098935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9921944-ddeb-4f19-903d-2f810b1a77d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:45.101332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9921944-ddeb-4f19-903d-2f810b1a77d4 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:45.275875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-448fe05b-645e-4e0c-98d1-281c89f20293 admin admin] The token cache is being invalidate because project d97ce3d33a254ff8b04aecd1c9e0b5f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:51:45.276528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-448fe05b-645e-4e0c-98d1-281c89f20293 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:45.277583 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-448fe05b-645e-4e0c-98d1-281c89f20293 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:51:45.279444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1118/2219] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:51:44 2022] POST /identity/v3/users => generated 295 bytes in 376 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:51:45.280692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9921944-ddeb-4f19-903d-2f810b1a77d4 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:45.283680 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b9921944-ddeb-4f19-903d-2f810b1a77d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:45.296304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1102/2220] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:51:44 2022] DELETE /identity/v3/projects/d97ce3d33a254ff8b04aecd1c9e0b5f6 => generated 0 bytes in 499 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:51:45.302378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b6995-081f-4ba7-87c8-e62c21d38561 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.303967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b6995-081f-4ba7-87c8-e62c21d38561 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.303967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b6995-081f-4ba7-87c8-e62c21d38561 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:45.314782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9921944-ddeb-4f19-903d-2f810b1a77d4 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:45.319328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:45.321642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555b459d-3159-4b90-8c65-4c9a3e1df518 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:45.323789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b459d-3159-4b90-8c65-4c9a3e1df518 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.324079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b459d-3159-4b90-8c65-4c9a3e1df518 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.324378 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b459d-3159-4b90-8c65-4c9a3e1df518 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:45.325060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555b459d-3159-4b90-8c65-4c9a3e1df518 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:45.326990 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555b459d-3159-4b90-8c65-4c9a3e1df518 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:45.343447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1119/2221] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:51:45 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:51:45.345909 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-612b6995-081f-4ba7-87c8-e62c21d38561 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:45.347303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-612b6995-081f-4ba7-87c8-e62c21d38561 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:45.347913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555b459d-3159-4b90-8c65-4c9a3e1df518 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:45.358391 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555b459d-3159-4b90-8c65-4c9a3e1df518 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:45.370178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1103/2222] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:45.370178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9394d3f4-452e-48be-8e13-a3036592b7c8 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.370178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9394d3f4-452e-48be-8e13-a3036592b7c8 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.370178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9394d3f4-452e-48be-8e13-a3036592b7c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:45.374614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c0dcc4-a92a-434b-9989-affadbb539fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:45.374614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c0dcc4-a92a-434b-9989-affadbb539fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.374614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c0dcc4-a92a-434b-9989-affadbb539fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.374614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c0dcc4-a92a-434b-9989-affadbb539fd admin admin] PATH_INFO: `/v3/projects/4af87005146c4e9b9af06274a0235eaa/users/711466756ea645b7846b473fdc76e490/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:45.395525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c0dcc4-a92a-434b-9989-affadbb539fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=4af87005146c4e9b9af06274a0235eaa, user_id=711466756ea645b7846b473fdc76e490, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:45.395992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-9394d3f4-452e-48be-8e13-a3036592b7c8 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:45.397376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-9394d3f4-452e-48be-8e13-a3036592b7c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:45.398386 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c0dcc4-a92a-434b-9989-affadbb539fd admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:45.410275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1104/2223] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:45.415033 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9394d3f4-452e-48be-8e13-a3036592b7c8 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:45.444945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9394d3f4-452e-48be-8e13-a3036592b7c8 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:45.444945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e6dca4-e03c-4f44-a342-2c58acf0d49b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:45.444945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e6dca4-e03c-4f44-a342-2c58acf0d49b admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.444945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e6dca4-e03c-4f44-a342-2c58acf0d49b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.445447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e6dca4-e03c-4f44-a342-2c58acf0d49b admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:45.445447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e6dca4-e03c-4f44-a342-2c58acf0d49b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:51:45.445447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e6dca4-e03c-4f44-a342-2c58acf0d49b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:45.445447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e6dca4-e03c-4f44-a342-2c58acf0d49b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:45.458593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1105/2224] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:51:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:51:45.463120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e6dca4-e03c-4f44-a342-2c58acf0d49b admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:45.475353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e6dca4-e03c-4f44-a342-2c58acf0d49b admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:45.481132 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2712c52-cc85-441d-99af-321bad275971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:45.482108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2712c52-cc85-441d-99af-321bad275971 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.482675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2712c52-cc85-441d-99af-321bad275971 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.483158 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2712c52-cc85-441d-99af-321bad275971 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:45.483990 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2712c52-cc85-441d-99af-321bad275971 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:45.487476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2712c52-cc85-441d-99af-321bad275971 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:45.524849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1120/2225] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:51:45 2022] PUT /identity/v3/projects/4af87005146c4e9b9af06274a0235eaa/users/711466756ea645b7846b473fdc76e490/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:51:45.528961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c0dcc4-a92a-434b-9989-affadbb539fd admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:45.604335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c0dcc4-a92a-434b-9989-affadbb539fd admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:45.613581 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08954a53-fba7-4ffd-b0a9-d79505c500e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:45.614685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08954a53-fba7-4ffd-b0a9-d79505c500e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.615145 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08954a53-fba7-4ffd-b0a9-d79505c500e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.615583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08954a53-fba7-4ffd-b0a9-d79505c500e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:45.616564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08954a53-fba7-4ffd-b0a9-d79505c500e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:45.618643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08954a53-fba7-4ffd-b0a9-d79505c500e4 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:45.630662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1121/2226] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:51:45 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:51:45.636345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08954a53-fba7-4ffd-b0a9-d79505c500e4 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:45.637320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1106/2227] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:51:45 2022] POST /identity/v3/projects => generated 373 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:51:45.638654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2712c52-cc85-441d-99af-321bad275971 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:45.648762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08954a53-fba7-4ffd-b0a9-d79505c500e4 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:45.712250 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2712c52-cc85-441d-99af-321bad275971 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:45.716553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653f4ebb-05c5-48be-bcbf-5a0e52914909 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:45.717194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653f4ebb-05c5-48be-bcbf-5a0e52914909 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.717474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653f4ebb-05c5-48be-bcbf-5a0e52914909 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.717752 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653f4ebb-05c5-48be-bcbf-5a0e52914909 admin admin] PATH_INFO: `/v3/projects/4af87005146c4e9b9af06274a0235eaa/users/711466756ea645b7846b473fdc76e490/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:45.719758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72b86a8-5d93-439c-8e6a-3d66f589114e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:45.720629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653f4ebb-05c5-48be-bcbf-5a0e52914909 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4af87005146c4e9b9af06274a0235eaa, user_id=711466756ea645b7846b473fdc76e490, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:45.721956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72b86a8-5d93-439c-8e6a-3d66f589114e admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.722428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72b86a8-5d93-439c-8e6a-3d66f589114e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.722841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72b86a8-5d93-439c-8e6a-3d66f589114e admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:45.723403 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653f4ebb-05c5-48be-bcbf-5a0e52914909 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:45.723828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72b86a8-5d93-439c-8e6a-3d66f589114e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:45.726220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72b86a8-5d93-439c-8e6a-3d66f589114e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:45.735840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1122/2228] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:51:45 2022] PUT /identity/v3/projects/4af87005146c4e9b9af06274a0235eaa/users/711466756ea645b7846b473fdc76e490/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:51:45.830647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1107/2229] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:51:45 2022] POST /identity/v3/users => generated 287 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:51:45.834432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653f4ebb-05c5-48be-bcbf-5a0e52914909 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:45.913532 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653f4ebb-05c5-48be-bcbf-5a0e52914909 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:45.918466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ff51b8-dd23-42e1-9d5d-15c7560c363f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:45.919065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ff51b8-dd23-42e1-9d5d-15c7560c363f admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.921197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ff51b8-dd23-42e1-9d5d-15c7560c363f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.921197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ff51b8-dd23-42e1-9d5d-15c7560c363f admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:45.921197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ff51b8-dd23-42e1-9d5d-15c7560c363f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:45.921795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ff51b8-dd23-42e1-9d5d-15c7560c363f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:45.932858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1123/2230] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:51:45 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:51:45.936736 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72b86a8-5d93-439c-8e6a-3d66f589114e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:45.949365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72b86a8-5d93-439c-8e6a-3d66f589114e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:45.954651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8953ca95-bb32-454d-b05f-7521229758e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:45.955267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8953ca95-bb32-454d-b05f-7521229758e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.955578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8953ca95-bb32-454d-b05f-7521229758e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.955851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8953ca95-bb32-454d-b05f-7521229758e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:45.956533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8953ca95-bb32-454d-b05f-7521229758e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:45.958135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8953ca95-bb32-454d-b05f-7521229758e9 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:45.968607 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1108/2231] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:51:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:51:45.973209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ff51b8-dd23-42e1-9d5d-15c7560c363f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:45.983642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ff51b8-dd23-42e1-9d5d-15c7560c363f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:45.989562 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2082ab6-7f54-4d4f-bf8f-b00d4351e242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:45.990417 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2082ab6-7f54-4d4f-bf8f-b00d4351e242 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:45.990736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2082ab6-7f54-4d4f-bf8f-b00d4351e242 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:45.991006 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2082ab6-7f54-4d4f-bf8f-b00d4351e242 admin admin] PATH_INFO: `/v3/projects/618d4ab58d984c78a68a69f393118927/users/2791e367943d4e1eaddd0f0f930ad8c1/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:46.006110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2082ab6-7f54-4d4f-bf8f-b00d4351e242 admin admin] RBAC: Authorizing `identity:create_grant(project_id=618d4ab58d984c78a68a69f393118927, user_id=2791e367943d4e1eaddd0f0f930ad8c1, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:46.009232 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2082ab6-7f54-4d4f-bf8f-b00d4351e242 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:46.133603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1124/2232] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:51:45 2022] PUT /identity/v3/projects/618d4ab58d984c78a68a69f393118927/users/2791e367943d4e1eaddd0f0f930ad8c1/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:51:46.145505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa527dac-fabc-4acd-a2f3-da61543fefa2 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:46.145794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa527dac-fabc-4acd-a2f3-da61543fefa2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:46.146071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa527dac-fabc-4acd-a2f3-da61543fefa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:46.222047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-aa527dac-fabc-4acd-a2f3-da61543fefa2 None None] MFA Rules not processed for user `2791e367943d4e1eaddd0f0f930ad8c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:46.223265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-aa527dac-fabc-4acd-a2f3-da61543fefa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:46.275479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:46.342282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1109/2233] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:46 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:46.357212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2082ab6-7f54-4d4f-bf8f-b00d4351e242 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:46.361547 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c2082ab6-7f54-4d4f-bf8f-b00d4351e242 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:46.503334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2082ab6-7f54-4d4f-bf8f-b00d4351e242 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:46.520730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2db34b-31b3-439c-998d-9e523c6d2833 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:46.521826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2db34b-31b3-439c-998d-9e523c6d2833 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:46.523124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2db34b-31b3-439c-998d-9e523c6d2833 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:46.523825 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2db34b-31b3-439c-998d-9e523c6d2833 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:46.538965 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2db34b-31b3-439c-998d-9e523c6d2833 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:46.546319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2db34b-31b3-439c-998d-9e523c6d2833 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:46.554683 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1125/2234] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:51:46 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:51:49.849393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a252c4-2d73-44b2-b2c1-d97b69161d7a None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:49.849393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a252c4-2d73-44b2-b2c1-d97b69161d7a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:49.850014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a252c4-2d73-44b2-b2c1-d97b69161d7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:49.882211 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-13a252c4-2d73-44b2-b2c1-d97b69161d7a None None] MFA Rules not processed for user `2791e367943d4e1eaddd0f0f930ad8c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:49.884541 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-13a252c4-2d73-44b2-b2c1-d97b69161d7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:49.899535 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1110/2235] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:49 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:54.289741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca4d7dd-d34c-4bf0-bc56-abfa5ecc9510 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:54.290335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca4d7dd-d34c-4bf0-bc56-abfa5ecc9510 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:54.290335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca4d7dd-d34c-4bf0-bc56-abfa5ecc9510 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:54.331176 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-7ca4d7dd-d34c-4bf0-bc56-abfa5ecc9510 None None] MFA Rules not processed for user `711466756ea645b7846b473fdc76e490`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:54.332528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7ca4d7dd-d34c-4bf0-bc56-abfa5ecc9510 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:54.401446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:54.475530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1126/2236] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:51:54 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:54.508759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a252c4-2d73-44b2-b2c1-d97b69161d7a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:54.509807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-13a252c4-2d73-44b2-b2c1-d97b69161d7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:54.586651 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:51:54.705784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a252c4-2d73-44b2-b2c1-d97b69161d7a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:54.712262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00b457c-7b73-4cc2-ac04-846a3e4fdd7c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:54.712900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00b457c-7b73-4cc2-ac04-846a3e4fdd7c service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:54.713169 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00b457c-7b73-4cc2-ac04-846a3e4fdd7c service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:54.713437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00b457c-7b73-4cc2-ac04-846a3e4fdd7c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:54.715448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a00b457c-7b73-4cc2-ac04-846a3e4fdd7c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:54.831429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00b457c-7b73-4cc2-ac04-846a3e4fdd7c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:54.833980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00b457c-7b73-4cc2-ac04-846a3e4fdd7c service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:54.846115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1111/2237] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:51:54 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:51:59.364065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c1f819-6e07-4c4d-8432-e17b52770ec7 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:59.365608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c1f819-6e07-4c4d-8432-e17b52770ec7 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:59.365608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c1f819-6e07-4c4d-8432-e17b52770ec7 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:59.366501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1127/2238] 10.222.0.65 () {54 vars in 921 bytes} [Wed Aug 24 03:51:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:51:59.376519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb30f6a-0080-43df-9d41-6bc239a54598 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:59.376519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb30f6a-0080-43df-9d41-6bc239a54598 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:59.377044 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb30f6a-0080-43df-9d41-6bc239a54598 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:59.413346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-8fb30f6a-0080-43df-9d41-6bc239a54598 None None] MFA Rules not processed for user `e5896018712d4b82a781365e478ee855`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:59.415108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-8fb30f6a-0080-43df-9d41-6bc239a54598 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:59.438317 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1112/2239] 10.222.0.65 () {58 vars in 1018 bytes} [Wed Aug 24 03:51:59 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:51:59.449281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c1f819-6e07-4c4d-8432-e17b52770ec7 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:51:59.451401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d5c1f819-6e07-4c4d-8432-e17b52770ec7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:59.530025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c1f819-6e07-4c4d-8432-e17b52770ec7 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:51:59.537627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb9d0b9-09f7-4df3-b7f9-a0ca434368b1 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f3b3c7ebc08496e970f8f25fd6395e4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:51:59.537627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb9d0b9-09f7-4df3-b7f9-a0ca434368b1 service glance] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:59.537627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb9d0b9-09f7-4df3-b7f9-a0ca434368b1 service glance] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:59.537627 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb9d0b9-09f7-4df3-b7f9-a0ca434368b1 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:59.548450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb9d0b9-09f7-4df3-b7f9-a0ca434368b1 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:51:59.549966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb9d0b9-09f7-4df3-b7f9-a0ca434368b1 service glance] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:51:59.563384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1128/2240] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:51:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:51:59.896533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f478ef86-eed5-4cb8-a428-6f081ce66bdd None None] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:59.896856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f478ef86-eed5-4cb8-a428-6f081ce66bdd None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:59.897118 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f478ef86-eed5-4cb8-a428-6f081ce66bdd None None] PATH_INFO: `/` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:59.898464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1113/2241] 10.222.0.65 () {54 vars in 921 bytes} [Wed Aug 24 03:51:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:51:59.909991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88ee430-537b-4966-9706-4960d409bd98 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:51:59.910258 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88ee430-537b-4966-9706-4960d409bd98 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:51:59.910686 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88ee430-537b-4966-9706-4960d409bd98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:51:59.938762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c88ee430-537b-4966-9706-4960d409bd98 None None] MFA Rules not processed for user `64f23f0c8abc406b9700361723623387`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:51:59.940045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c88ee430-537b-4966-9706-4960d409bd98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:51:59.994570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1129/2242] 10.222.0.65 () {58 vars in 1018 bytes} [Wed Aug 24 03:51:59 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:52:00.007640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f478ef86-eed5-4cb8-a428-6f081ce66bdd None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:52:00.009424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f478ef86-eed5-4cb8-a428-6f081ce66bdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:52:00.025013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f478ef86-eed5-4cb8-a428-6f081ce66bdd None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:52:00.029368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3464eb4-f37f-4571-9a6e-484c771b3d28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:52:00.029972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3464eb4-f37f-4571-9a6e-484c771b3d28 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:00.030273 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3464eb4-f37f-4571-9a6e-484c771b3d28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:00.030562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3464eb4-f37f-4571-9a6e-484c771b3d28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:00.039063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3464eb4-f37f-4571-9a6e-484c771b3d28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:52:00.040743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3464eb4-f37f-4571-9a6e-484c771b3d28 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:52:00.051659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1114/2243] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:52:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:52:00.136888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d18feb-158c-4726-94d6-235340c68567 None None] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:00.137206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d18feb-158c-4726-94d6-235340c68567 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:00.137486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d18feb-158c-4726-94d6-235340c68567 None None] PATH_INFO: `/` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:00.139075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1130/2244] 10.222.0.65 () {54 vars in 921 bytes} [Wed Aug 24 03:52:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 24 03:52:00.144574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4049d5-31a8-4cfa-ad22-f7d541154f99 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:00.144885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4049d5-31a8-4cfa-ad22-f7d541154f99 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:00.145188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4049d5-31a8-4cfa-ad22-f7d541154f99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:00.166175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-bb4049d5-31a8-4cfa-ad22-f7d541154f99 None None] MFA Rules not processed for user `e5896018712d4b82a781365e478ee855`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:52:00.167452 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-bb4049d5-31a8-4cfa-ad22-f7d541154f99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:52:00.177797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1115/2245] 10.222.0.65 () {58 vars in 1018 bytes} [Wed Aug 24 03:52:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:52:00.188912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d18feb-158c-4726-94d6-235340c68567 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:52:00.200103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d18feb-158c-4726-94d6-235340c68567 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:52:00.207714 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45836a2b-038e-45c0-91a5-7e55aad9d0b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:52:00.207714 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45836a2b-038e-45c0-91a5-7e55aad9d0b2 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:00.207714 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45836a2b-038e-45c0-91a5-7e55aad9d0b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:00.207714 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45836a2b-038e-45c0-91a5-7e55aad9d0b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:00.215784 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45836a2b-038e-45c0-91a5-7e55aad9d0b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:52:00.217217 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45836a2b-038e-45c0-91a5-7e55aad9d0b2 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:52:00.229503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1131/2246] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:52:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:52:10.161063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=61 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:52:10.224295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=63 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:52:28.454001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efd6e13-61f2-45b0-865d-745cdfaa02c2 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:28.454829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efd6e13-61f2-45b0-865d-745cdfaa02c2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:28.455108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efd6e13-61f2-45b0-865d-745cdfaa02c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:28.477082 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-3efd6e13-61f2-45b0-865d-745cdfaa02c2 None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:52:28.478409 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3efd6e13-61f2-45b0-865d-745cdfaa02c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:52:28.566294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1116/2247] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:52:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:52:28.947741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45836a2b-038e-45c0-91a5-7e55aad9d0b2 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:52:28.948702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-45836a2b-038e-45c0-91a5-7e55aad9d0b2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:52:29.026758 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45836a2b-038e-45c0-91a5-7e55aad9d0b2 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:52:29.033421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b0ff5d-def0-45b6-bc5e-c1122040f840 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:52:29.033421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b0ff5d-def0-45b6-bc5e-c1122040f840 service swift] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:29.033685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b0ff5d-def0-45b6-bc5e-c1122040f840 service swift] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:29.033960 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b0ff5d-def0-45b6-bc5e-c1122040f840 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:29.042693 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b0ff5d-def0-45b6-bc5e-c1122040f840 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:52:29.044282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b0ff5d-def0-45b6-bc5e-c1122040f840 service swift] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:52:29.052744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1132/2248] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:52:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:52:29.523552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afb9069-2593-4515-bbe1-f85521e98151 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:29.524587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afb9069-2593-4515-bbe1-f85521e98151 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:29.525087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afb9069-2593-4515-bbe1-f85521e98151 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:29.553121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-5afb9069-2593-4515-bbe1-f85521e98151 None None] MFA Rules not processed for user `71ff73cade8b42c7ae57be1e3ef0e028`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:52:29.554398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5afb9069-2593-4515-bbe1-f85521e98151 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:52:29.562823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1117/2249] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:52:29 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:52:39.049406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=32 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:52:39.545175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=28 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:52:56.063293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588fa966-7d8a-43b2-969a-786101f75d9c None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:56.063293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588fa966-7d8a-43b2-969a-786101f75d9c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:56.063293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588fa966-7d8a-43b2-969a-786101f75d9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:56.082942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-588fa966-7d8a-43b2-969a-786101f75d9c None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:52:56.084094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-588fa966-7d8a-43b2-969a-786101f75d9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:52:56.094026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1133/2250] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:52:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:52:56.106512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afb9069-2593-4515-bbe1-f85521e98151 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:52:56.121374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afb9069-2593-4515-bbe1-f85521e98151 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:52:56.127478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79844599-1abb-4105-9132-3b1273e5b5f8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:52:56.128104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79844599-1abb-4105-9132-3b1273e5b5f8 service swift] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:56.128381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79844599-1abb-4105-9132-3b1273e5b5f8 service swift] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:56.128657 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79844599-1abb-4105-9132-3b1273e5b5f8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:56.137886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79844599-1abb-4105-9132-3b1273e5b5f8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:52:56.139596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79844599-1abb-4105-9132-3b1273e5b5f8 service swift] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:52:56.148446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1118/2251] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:52:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:52:57.024693 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f3d8e-6e02-42d2-9692-42da77fe5b21 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:57.025301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f3d8e-6e02-42d2-9692-42da77fe5b21 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:57.025301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f3d8e-6e02-42d2-9692-42da77fe5b21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:57.044824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-ed8f3d8e-6e02-42d2-9692-42da77fe5b21 None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:52:57.046598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ed8f3d8e-6e02-42d2-9692-42da77fe5b21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:52:57.053769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1134/2252] 10.222.0.65 () {58 vars in 967 bytes} [Wed Aug 24 03:52:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:52:57.067134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79844599-1abb-4105-9132-3b1273e5b5f8 service swift] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:52:57.078443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79844599-1abb-4105-9132-3b1273e5b5f8 service swift] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:52:57.085411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e596ed9a-713e-4ef9-b652-b5e98f770ec7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:52:57.086101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e596ed9a-713e-4ef9-b652-b5e98f770ec7 service swift] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:57.087522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e596ed9a-713e-4ef9-b652-b5e98f770ec7 service swift] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:57.087522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e596ed9a-713e-4ef9-b652-b5e98f770ec7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:57.096963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e596ed9a-713e-4ef9-b652-b5e98f770ec7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:52:57.098599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e596ed9a-713e-4ef9-b652-b5e98f770ec7 service swift] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:52:57.107415 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1119/2253] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:52:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:52:57.989736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dc1392-31c4-4f7a-b9c9-623d125834c2 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:57.990388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dc1392-31c4-4f7a-b9c9-623d125834c2 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:57.990388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dc1392-31c4-4f7a-b9c9-623d125834c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:58.011555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-72dc1392-31c4-4f7a-b9c9-623d125834c2 None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:52:58.015022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-72dc1392-31c4-4f7a-b9c9-623d125834c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:52:58.021027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1135/2254] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:52:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:52:58.033952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e596ed9a-713e-4ef9-b652-b5e98f770ec7 service swift] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:52:58.044548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e596ed9a-713e-4ef9-b652-b5e98f770ec7 service swift] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:52:58.051018 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6413e87-ce19-4483-81dc-5b916dbc96e1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:52:58.051681 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6413e87-ce19-4483-81dc-5b916dbc96e1 service swift] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:58.051896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6413e87-ce19-4483-81dc-5b916dbc96e1 service swift] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:58.052162 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6413e87-ce19-4483-81dc-5b916dbc96e1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:58.061617 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6413e87-ce19-4483-81dc-5b916dbc96e1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:52:58.063382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6413e87-ce19-4483-81dc-5b916dbc96e1 service swift] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:52:58.072365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1120/2255] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:52:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:52:58.771893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4bb9f8-3a9f-47dc-9c3a-ff621157b077 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:58.772202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4bb9f8-3a9f-47dc-9c3a-ff621157b077 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:58.772471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4bb9f8-3a9f-47dc-9c3a-ff621157b077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:58.792552 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1d4bb9f8-3a9f-47dc-9c3a-ff621157b077 None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:52:58.793777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1d4bb9f8-3a9f-47dc-9c3a-ff621157b077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:52:58.801939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1136/2256] 10.222.0.65 () {58 vars in 967 bytes} [Wed Aug 24 03:52:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:52:58.816116 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6413e87-ce19-4483-81dc-5b916dbc96e1 service swift] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:52:58.826974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6413e87-ce19-4483-81dc-5b916dbc96e1 service swift] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:52:58.832453 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133bfcd9-7a54-45d1-b7d3-833938f68e40 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:52:58.833045 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133bfcd9-7a54-45d1-b7d3-833938f68e40 service swift] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:52:58.833348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133bfcd9-7a54-45d1-b7d3-833938f68e40 service swift] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:52:58.833619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133bfcd9-7a54-45d1-b7d3-833938f68e40 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:52:58.841699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133bfcd9-7a54-45d1-b7d3-833938f68e40 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:52:58.844017 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133bfcd9-7a54-45d1-b7d3-833938f68e40 service swift] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:52:58.852214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1121/2257] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:52:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:53:01.069892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b80753-8264-4a4c-9370-59ed25710faa None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:01.070402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b80753-8264-4a4c-9370-59ed25710faa None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:01.070524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b80753-8264-4a4c-9370-59ed25710faa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:01.095480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-09b80753-8264-4a4c-9370-59ed25710faa None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:53:01.097257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-09b80753-8264-4a4c-9370-59ed25710faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:01.108130 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1137/2258] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:53:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:53:01.128067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133bfcd9-7a54-45d1-b7d3-833938f68e40 service swift] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:01.148218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133bfcd9-7a54-45d1-b7d3-833938f68e40 service swift] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:01.156746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb716afc-be37-476d-9c82-c147cf03337b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:01.162732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb716afc-be37-476d-9c82-c147cf03337b service swift] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:01.163709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb716afc-be37-476d-9c82-c147cf03337b service swift] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:01.171251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb716afc-be37-476d-9c82-c147cf03337b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:01.200345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb716afc-be37-476d-9c82-c147cf03337b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:01.207829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb716afc-be37-476d-9c82-c147cf03337b service swift] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:01.218606 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1122/2259] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:53:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:53:06.572527 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b80753-8264-4a4c-9370-59ed25710faa None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:06.657334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b80753-8264-4a4c-9370-59ed25710faa None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:06.662609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:06.663257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:06.663533 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:06.663801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] PATH_INFO: `/v3/users/2791e367943d4e1eaddd0f0f930ad8c1` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:06.665380 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] RBAC: Authorizing `identity:delete_user(user_id=2791e367943d4e1eaddd0f0f930ad8c1)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:06.667628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:06.899585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2791e367943d4e1eaddd0f0f930ad8c1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:06.900783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2791e367943d4e1eaddd0f0f930ad8c1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:06.912461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2791e367943d4e1eaddd0f0f930ad8c1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:06.914052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2791e367943d4e1eaddd0f0f930ad8c1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:07.007885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2791e367943d4e1eaddd0f0f930ad8c1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:07.034087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1138/2260] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:53:06 2022] DELETE /identity/v3/users/2791e367943d4e1eaddd0f0f930ad8c1 => generated 0 bytes in 462 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:53:07.038979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dfa9c5-5be7-4059-924d-edb6551c7add None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:07.040154 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dfa9c5-5be7-4059-924d-edb6551c7add None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:07.040154 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dfa9c5-5be7-4059-924d-edb6551c7add None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:07.072832 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-c2dfa9c5-5be7-4059-924d-edb6551c7add None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:53:07.075202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c2dfa9c5-5be7-4059-924d-edb6551c7add None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:07.191652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1123/2261] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:53:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:53:07.203899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:07.206849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:07.277214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:53:07.285209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4303f5f8-9cad-4f68-bd07-b2c5752bc52b admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:07.289513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557caebd-dd37-4446-afd9-29bf0e715a9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:07.290218 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557caebd-dd37-4446-afd9-29bf0e715a9b service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:07.290513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557caebd-dd37-4446-afd9-29bf0e715a9b service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:07.290776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557caebd-dd37-4446-afd9-29bf0e715a9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:07.303216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557caebd-dd37-4446-afd9-29bf0e715a9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:07.305415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557caebd-dd37-4446-afd9-29bf0e715a9b service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:07.319169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1139/2262] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:53:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:53:07.682870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2dfa9c5-5be7-4059-924d-edb6551c7add None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:07.686324 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c2dfa9c5-5be7-4059-924d-edb6551c7add None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:07.729534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2dfa9c5-5be7-4059-924d-edb6551c7add None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:07.745211 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:07.746930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:07.747475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:07.748557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] PATH_INFO: `/v3/projects/618d4ab58d984c78a68a69f393118927` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:07.751061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] RBAC: Authorizing `identity:delete_project(project_id=618d4ab58d984c78a68a69f393118927)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:07.771019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:07.989857 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '618d4ab58d984c78a68a69f393118927', 'request_id': 'req-eaec44a1-7682-43d6-bb05-2436003d882b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:08.010926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '618d4ab58d984c78a68a69f393118927', 'request_id': 'req-eaec44a1-7682-43d6-bb05-2436003d882b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:08.021674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '618d4ab58d984c78a68a69f393118927', 'request_id': 'req-eaec44a1-7682-43d6-bb05-2436003d882b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:08.129618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] The token cache is being invalidate because project 618d4ab58d984c78a68a69f393118927 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:53:08.130340 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:08.131760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:08.135180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1124/2263] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:53:07 2022] DELETE /identity/v3/projects/618d4ab58d984c78a68a69f393118927 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:53:13.349135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72255c36-bd98-4ea5-b64a-8cb8f2b42cc4 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:13.349690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72255c36-bd98-4ea5-b64a-8cb8f2b42cc4 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:13.349793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72255c36-bd98-4ea5-b64a-8cb8f2b42cc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:13.370264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-72255c36-bd98-4ea5-b64a-8cb8f2b42cc4 None None] MFA Rules not processed for user `5247fd6c30794dc48f60a827d5837674`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:53:13.371495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-72255c36-bd98-4ea5-b64a-8cb8f2b42cc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:13.466273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1140/2264] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:53:13 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:53:14.138191 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:14.139746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:14.165412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:53:14.225180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaec44a1-7682-43d6-bb05-2436003d882b admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:14.229826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834a16c6-4030-4b10-a5f7-e37d4e6927a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:14.237189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834a16c6-4030-4b10-a5f7-e37d4e6927a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:14.237189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834a16c6-4030-4b10-a5f7-e37d4e6927a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:14.237189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834a16c6-4030-4b10-a5f7-e37d4e6927a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:14.237189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-834a16c6-4030-4b10-a5f7-e37d4e6927a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:14.336038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834a16c6-4030-4b10-a5f7-e37d4e6927a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:14.337750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834a16c6-4030-4b10-a5f7-e37d4e6927a5 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:14.354406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1125/2265] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:53:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:53:16.015574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4f3fe3-6526-4628-a7a7-adc8bf6fc105 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:16.016166 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4f3fe3-6526-4628-a7a7-adc8bf6fc105 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:16.021697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4f3fe3-6526-4628-a7a7-adc8bf6fc105 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:16.046904 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1f4f3fe3-6526-4628-a7a7-adc8bf6fc105 None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:53:16.048481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1f4f3fe3-6526-4628-a7a7-adc8bf6fc105 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:16.130501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1141/2266] 10.222.0.65 () {58 vars in 967 bytes} [Wed Aug 24 03:53:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:53:16.154925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834a16c6-4030-4b10-a5f7-e37d4e6927a5 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:16.158638 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-834a16c6-4030-4b10-a5f7-e37d4e6927a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:16.277106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834a16c6-4030-4b10-a5f7-e37d4e6927a5 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:16.284426 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb06b22-583f-4c79-9dc9-90cda7bbb4d6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:16.285213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb06b22-583f-4c79-9dc9-90cda7bbb4d6 service swift] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:16.285566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb06b22-583f-4c79-9dc9-90cda7bbb4d6 service swift] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:16.285933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb06b22-583f-4c79-9dc9-90cda7bbb4d6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:16.300343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb06b22-583f-4c79-9dc9-90cda7bbb4d6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:16.302500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb06b22-583f-4c79-9dc9-90cda7bbb4d6 service swift] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:16.320721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1126/2267] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:53:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:53:16.872601 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4f3fe3-6526-4628-a7a7-adc8bf6fc105 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:16.883566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4f3fe3-6526-4628-a7a7-adc8bf6fc105 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:16.887298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273fe748-7a27-4888-a4d5-c1634bedcf23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:16.887839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273fe748-7a27-4888-a4d5-c1634bedcf23 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:16.888235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273fe748-7a27-4888-a4d5-c1634bedcf23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:16.888463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273fe748-7a27-4888-a4d5-c1634bedcf23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:16.890589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-273fe748-7a27-4888-a4d5-c1634bedcf23 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:16.906339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273fe748-7a27-4888-a4d5-c1634bedcf23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:16.907859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273fe748-7a27-4888-a4d5-c1634bedcf23 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:16.921896 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1142/2268] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:53:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:53:23.992230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb06b22-583f-4c79-9dc9-90cda7bbb4d6 service swift] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:23.993669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0cb06b22-583f-4c79-9dc9-90cda7bbb4d6 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:24.006582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb06b22-583f-4c79-9dc9-90cda7bbb4d6 service swift] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:24.010485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe1269e-9bfa-489f-bda8-fd53458d85da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:24.011084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe1269e-9bfa-489f-bda8-fd53458d85da service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:24.011339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe1269e-9bfa-489f-bda8-fd53458d85da service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:24.011593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe1269e-9bfa-489f-bda8-fd53458d85da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:24.014324 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7fe1269e-9bfa-489f-bda8-fd53458d85da service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:24.017493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb6939-b532-4438-8333-df35a1d391ce None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:24.018022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb6939-b532-4438-8333-df35a1d391ce None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:24.018445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb6939-b532-4438-8333-df35a1d391ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:24.041527 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-eefb6939-b532-4438-8333-df35a1d391ce None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:53:24.042789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-eefb6939-b532-4438-8333-df35a1d391ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:24.050248 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1143/2269] 10.222.0.65 () {58 vars in 967 bytes} [Wed Aug 24 03:53:24 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:53:24.067493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:53:24.071287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefb6939-b532-4438-8333-df35a1d391ce None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:24.087954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefb6939-b532-4438-8333-df35a1d391ce None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:24.095787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ec9221-44e9-4190-8e43-8e1343fadcec service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:24.096845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ec9221-44e9-4190-8e43-8e1343fadcec service swift] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:24.097282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ec9221-44e9-4190-8e43-8e1343fadcec service swift] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:24.097396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe1269e-9bfa-489f-bda8-fd53458d85da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:24.097710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ec9221-44e9-4190-8e43-8e1343fadcec service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:24.098899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe1269e-9bfa-489f-bda8-fd53458d85da service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:24.110787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1127/2270] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:53:23 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:53:24.112196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ec9221-44e9-4190-8e43-8e1343fadcec service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:24.114037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ec9221-44e9-4190-8e43-8e1343fadcec service swift] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:24.124324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1144/2271] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:53:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:53:28.708859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe1269e-9bfa-489f-bda8-fd53458d85da service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:28.711047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7fe1269e-9bfa-489f-bda8-fd53458d85da service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:28.808463 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe1269e-9bfa-489f-bda8-fd53458d85da service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:28.813893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c2a8ea-57bd-4648-bb6e-f8c6d6c45a23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:28.815210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c2a8ea-57bd-4648-bb6e-f8c6d6c45a23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:28.815682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c2a8ea-57bd-4648-bb6e-f8c6d6c45a23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:28.816355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c2a8ea-57bd-4648-bb6e-f8c6d6c45a23 admin admin] PATH_INFO: `/v3/users/ae04cdfc35964cf58502cefd1cc912e9` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:28.817780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c2a8ea-57bd-4648-bb6e-f8c6d6c45a23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae04cdfc35964cf58502cefd1cc912e9)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:28.820047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c2a8ea-57bd-4648-bb6e-f8c6d6c45a23 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:29.124546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-f7c2a8ea-57bd-4648-bb6e-f8c6d6c45a23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae04cdfc35964cf58502cefd1cc912e9'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:29.125261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-f7c2a8ea-57bd-4648-bb6e-f8c6d6c45a23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae04cdfc35964cf58502cefd1cc912e9'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:29.135507 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-f7c2a8ea-57bd-4648-bb6e-f8c6d6c45a23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae04cdfc35964cf58502cefd1cc912e9'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:29.148658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-f7c2a8ea-57bd-4648-bb6e-f8c6d6c45a23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae04cdfc35964cf58502cefd1cc912e9'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:29.284671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-f7c2a8ea-57bd-4648-bb6e-f8c6d6c45a23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae04cdfc35964cf58502cefd1cc912e9'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:29.293219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1128/2272] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:53:28 2022] DELETE /identity/v3/users/ae04cdfc35964cf58502cefd1cc912e9 => generated 0 bytes in 585 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:53:29.582076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ec9221-44e9-4190-8e43-8e1343fadcec service swift] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:29.593195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ec9221-44e9-4190-8e43-8e1343fadcec service swift] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:29.657187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868da78a-7a48-441d-8414-7d568ee3cdf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:29.657833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868da78a-7a48-441d-8414-7d568ee3cdf5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:29.658114 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868da78a-7a48-441d-8414-7d568ee3cdf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:29.658435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868da78a-7a48-441d-8414-7d568ee3cdf5 admin admin] PATH_INFO: `/v3/projects/3cf3b9526fc541048f921613ded8ef2d` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:29.659588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868da78a-7a48-441d-8414-7d568ee3cdf5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3cf3b9526fc541048f921613ded8ef2d)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:29.661513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868da78a-7a48-441d-8414-7d568ee3cdf5 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:29.670609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:53:29.799310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-868da78a-7a48-441d-8414-7d568ee3cdf5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3cf3b9526fc541048f921613ded8ef2d', 'request_id': 'req-868da78a-7a48-441d-8414-7d568ee3cdf5'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:29.808734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-868da78a-7a48-441d-8414-7d568ee3cdf5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3cf3b9526fc541048f921613ded8ef2d', 'request_id': 'req-868da78a-7a48-441d-8414-7d568ee3cdf5'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:29.893690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-868da78a-7a48-441d-8414-7d568ee3cdf5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3cf3b9526fc541048f921613ded8ef2d', 'request_id': 'req-868da78a-7a48-441d-8414-7d568ee3cdf5'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:29.907583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-868da78a-7a48-441d-8414-7d568ee3cdf5 admin admin] The token cache is being invalidate because project 3cf3b9526fc541048f921613ded8ef2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:53:29.909035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-868da78a-7a48-441d-8414-7d568ee3cdf5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:29.909035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-868da78a-7a48-441d-8414-7d568ee3cdf5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:29.910108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1145/2273] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:53:29 2022] DELETE /identity/v3/projects/3cf3b9526fc541048f921613ded8ef2d => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:53:29.923345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a1756d-16c7-4be7-9a82-1cd6af0d23a8 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:29.924067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a1756d-16c7-4be7-9a82-1cd6af0d23a8 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:29.924355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a1756d-16c7-4be7-9a82-1cd6af0d23a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:29.946285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-13a1756d-16c7-4be7-9a82-1cd6af0d23a8 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:53:29.947563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-13a1756d-16c7-4be7-9a82-1cd6af0d23a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:30.017657 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1129/2274] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:53:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:53:30.025286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8aa61d-c768-4581-8e7e-4b586aa93c85 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:30.025575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8aa61d-c768-4581-8e7e-4b586aa93c85 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:30.025853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8aa61d-c768-4581-8e7e-4b586aa93c85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:30.048569 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-dc8aa61d-c768-4581-8e7e-4b586aa93c85 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:53:30.049891 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-dc8aa61d-c768-4581-8e7e-4b586aa93c85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:30.058254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1146/2275] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:53:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:53:30.063578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a1756d-16c7-4be7-9a82-1cd6af0d23a8 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:30.068829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:53:30.074976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a1756d-16c7-4be7-9a82-1cd6af0d23a8 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:30.080082 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e204e34-c7cd-4823-8fbf-0617524c599f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:30.080745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e204e34-c7cd-4823-8fbf-0617524c599f admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:30.081028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e204e34-c7cd-4823-8fbf-0617524c599f admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:30.081295 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e204e34-c7cd-4823-8fbf-0617524c599f admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:30.081972 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e204e34-c7cd-4823-8fbf-0617524c599f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:53:30.082327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e204e34-c7cd-4823-8fbf-0617524c599f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:30.083925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e204e34-c7cd-4823-8fbf-0617524c599f admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:30.096264 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8aa61d-c768-4581-8e7e-4b586aa93c85 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:30.097670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-dc8aa61d-c768-4581-8e7e-4b586aa93c85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:30.097942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1130/2276] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:53:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:53:30.102066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e204e34-c7cd-4823-8fbf-0617524c599f admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:30.115520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e204e34-c7cd-4823-8fbf-0617524c599f admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:30.121346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b0910d-43bc-4822-8709-f5abbd3cf6aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:30.124496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b0910d-43bc-4822-8709-f5abbd3cf6aa admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:30.125166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b0910d-43bc-4822-8709-f5abbd3cf6aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:30.125446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b0910d-43bc-4822-8709-f5abbd3cf6aa admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:30.126232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b0910d-43bc-4822-8709-f5abbd3cf6aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:30.128257 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b0910d-43bc-4822-8709-f5abbd3cf6aa admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:30.200800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8aa61d-c768-4581-8e7e-4b586aa93c85 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:30.207623 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437ecd82-8f28-4fac-a288-563ef701c3b5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:30.208490 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437ecd82-8f28-4fac-a288-563ef701c3b5 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:30.208937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437ecd82-8f28-4fac-a288-563ef701c3b5 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:30.209214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437ecd82-8f28-4fac-a288-563ef701c3b5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:30.211345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-437ecd82-8f28-4fac-a288-563ef701c3b5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:30.243253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1131/2277] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:53:30 2022] POST /identity/v3/projects => generated 371 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:53:30.247250 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b0910d-43bc-4822-8709-f5abbd3cf6aa admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:30.291549 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437ecd82-8f28-4fac-a288-563ef701c3b5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:30.293146 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437ecd82-8f28-4fac-a288-563ef701c3b5 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:30.305831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1147/2278] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:53:30 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:53:30.339757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b0910d-43bc-4822-8709-f5abbd3cf6aa admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:30.345001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3058356b-37ec-4603-acfd-25ce38e4a67a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:30.345625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3058356b-37ec-4603-acfd-25ce38e4a67a admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:30.345904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3058356b-37ec-4603-acfd-25ce38e4a67a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:30.346378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3058356b-37ec-4603-acfd-25ce38e4a67a admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:30.347059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3058356b-37ec-4603-acfd-25ce38e4a67a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:30.348998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3058356b-37ec-4603-acfd-25ce38e4a67a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:30.447608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1132/2279] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:53:30 2022] POST /identity/v3/users => generated 286 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:53:30.452367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437ecd82-8f28-4fac-a288-563ef701c3b5 service nova] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:30.463215 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437ecd82-8f28-4fac-a288-563ef701c3b5 service nova] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:30.468138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b6353-8659-4313-8600-4e9727b76ed0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:30.468755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b6353-8659-4313-8600-4e9727b76ed0 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:30.469038 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b6353-8659-4313-8600-4e9727b76ed0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:30.469312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b6353-8659-4313-8600-4e9727b76ed0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:30.469963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9b6353-8659-4313-8600-4e9727b76ed0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:30.471602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9b6353-8659-4313-8600-4e9727b76ed0 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:30.483353 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1148/2280] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:53:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:53:30.487482 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3058356b-37ec-4603-acfd-25ce38e4a67a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:30.498427 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3058356b-37ec-4603-acfd-25ce38e4a67a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:30.503190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f3c7a8-284c-411e-b50f-db46e430f6a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:30.503790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f3c7a8-284c-411e-b50f-db46e430f6a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:30.504067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f3c7a8-284c-411e-b50f-db46e430f6a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:30.504337 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f3c7a8-284c-411e-b50f-db46e430f6a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:30.504978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f3c7a8-284c-411e-b50f-db46e430f6a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:30.506653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f3c7a8-284c-411e-b50f-db46e430f6a9 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:30.519639 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1133/2281] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:53:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:53:30.524164 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b6353-8659-4313-8600-4e9727b76ed0 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:30.534605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b6353-8659-4313-8600-4e9727b76ed0 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:30.539495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d01ca0-4d87-4f4e-8945-fd807d5cfd1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:30.540172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d01ca0-4d87-4f4e-8945-fd807d5cfd1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:30.540450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d01ca0-4d87-4f4e-8945-fd807d5cfd1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:30.540724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d01ca0-4d87-4f4e-8945-fd807d5cfd1b admin admin] PATH_INFO: `/v3/projects/d7cd301e2896414c8ca22a11787b7c0a/users/1d7271805d06456a90fc659939b97c1b/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:30.554644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:53:30.561016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d01ca0-4d87-4f4e-8945-fd807d5cfd1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7cd301e2896414c8ca22a11787b7c0a, user_id=1d7271805d06456a90fc659939b97c1b, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:30.564224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d01ca0-4d87-4f4e-8945-fd807d5cfd1b admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:30.691131 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1149/2282] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:53:30 2022] PUT /identity/v3/projects/d7cd301e2896414c8ca22a11787b7c0a/users/1d7271805d06456a90fc659939b97c1b/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:53:30.702931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa945f5-0654-4b8b-904b-8a517cf2b2a2 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:30.703261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa945f5-0654-4b8b-904b-8a517cf2b2a2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:30.703564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa945f5-0654-4b8b-904b-8a517cf2b2a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:30.733576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-cfa945f5-0654-4b8b-904b-8a517cf2b2a2 None None] MFA Rules not processed for user `1d7271805d06456a90fc659939b97c1b`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:53:30.734816 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-cfa945f5-0654-4b8b-904b-8a517cf2b2a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:30.802507 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1134/2283] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:53:30 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:53:30.816235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d01ca0-4d87-4f4e-8945-fd807d5cfd1b admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:30.817905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c0d01ca0-4d87-4f4e-8945-fd807d5cfd1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:30.878845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d01ca0-4d87-4f4e-8945-fd807d5cfd1b admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:30.882954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9d8a10-f9a5-4cff-bf9f-21930e09940b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:30.883564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9d8a10-f9a5-4cff-bf9f-21930e09940b service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:30.883846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9d8a10-f9a5-4cff-bf9f-21930e09940b service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:30.884119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9d8a10-f9a5-4cff-bf9f-21930e09940b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:30.893663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9d8a10-f9a5-4cff-bf9f-21930e09940b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:30.895175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9d8a10-f9a5-4cff-bf9f-21930e09940b service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:30.904924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1150/2284] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:53:30 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:53:30.932656 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa945f5-0654-4b8b-904b-8a517cf2b2a2 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:30.976460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:53:31.001370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa945f5-0654-4b8b-904b-8a517cf2b2a2 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:31.006237 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:31.007183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:31.007546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:31.008288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] PATH_INFO: `/v3/users/1d7271805d06456a90fc659939b97c1b` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:31.009641 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d7271805d06456a90fc659939b97c1b)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:31.011750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:31.215022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d7271805d06456a90fc659939b97c1b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:31.215695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d7271805d06456a90fc659939b97c1b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:31.223532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d7271805d06456a90fc659939b97c1b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:31.235749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d7271805d06456a90fc659939b97c1b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:31.303789 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d7271805d06456a90fc659939b97c1b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:31.310389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1135/2285] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:53:30 2022] DELETE /identity/v3/users/1d7271805d06456a90fc659939b97c1b => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:53:31.315679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70933c9f-ff44-45c1-b2e5-a849147eb27d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:31.316043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70933c9f-ff44-45c1-b2e5-a849147eb27d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:31.316356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70933c9f-ff44-45c1-b2e5-a849147eb27d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:31.336423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-70933c9f-ff44-45c1-b2e5-a849147eb27d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:53:31.337628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-70933c9f-ff44-45c1-b2e5-a849147eb27d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:31.415063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1151/2286] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:53:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:53:31.423790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:31.425634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:31.513837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1f1f6d-fb53-4d41-baa0-447783ceb22e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:31.517746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc4aabc-bac7-4510-8a8c-a55c27557abe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:31.518430 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc4aabc-bac7-4510-8a8c-a55c27557abe service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:31.518707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc4aabc-bac7-4510-8a8c-a55c27557abe service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:31.518994 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc4aabc-bac7-4510-8a8c-a55c27557abe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:31.529266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc4aabc-bac7-4510-8a8c-a55c27557abe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:31.530793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc4aabc-bac7-4510-8a8c-a55c27557abe service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:31.541124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1136/2287] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:53:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:53:31.695344 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70933c9f-ff44-45c1-b2e5-a849147eb27d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:31.696853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-70933c9f-ff44-45c1-b2e5-a849147eb27d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:31.771320 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70933c9f-ff44-45c1-b2e5-a849147eb27d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:31.776974 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f137d979-645f-46f5-a4be-94a63bc7a7e8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:31.777583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f137d979-645f-46f5-a4be-94a63bc7a7e8 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:31.777861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f137d979-645f-46f5-a4be-94a63bc7a7e8 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:31.778132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f137d979-645f-46f5-a4be-94a63bc7a7e8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:31.780503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f137d979-645f-46f5-a4be-94a63bc7a7e8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:31.815417 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:53:31.852027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f137d979-645f-46f5-a4be-94a63bc7a7e8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:31.853712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f137d979-645f-46f5-a4be-94a63bc7a7e8 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:31.865342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1152/2288] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:53:31 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:53:32.109772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc4aabc-bac7-4510-8a8c-a55c27557abe service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:32.111284 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6dc4aabc-bac7-4510-8a8c-a55c27557abe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:32.125579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc4aabc-bac7-4510-8a8c-a55c27557abe service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:32.130413 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:32.131063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:32.131332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:32.131599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3 admin admin] PATH_INFO: `/v3/projects/d7cd301e2896414c8ca22a11787b7c0a` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:32.132758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d7cd301e2896414c8ca22a11787b7c0a)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:32.135372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:32.258501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7cd301e2896414c8ca22a11787b7c0a', 'request_id': 'req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:32.260516 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:53:32.264914 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7cd301e2896414c8ca22a11787b7c0a', 'request_id': 'req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:32.271223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7cd301e2896414c8ca22a11787b7c0a', 'request_id': 'req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:32.337331 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3 admin admin] The token cache is being invalidate because project d7cd301e2896414c8ca22a11787b7c0a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:53:32.337623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:32.338339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ced2d50a-fcc1-4c81-b97b-171efcfd34a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:53:32.340214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1137/2289] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:53:32 2022] DELETE /identity/v3/projects/d7cd301e2896414c8ca22a11787b7c0a => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:53:35.793801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f137d979-645f-46f5-a4be-94a63bc7a7e8 service nova] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:35.795564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f137d979-645f-46f5-a4be-94a63bc7a7e8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:35.823795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f137d979-645f-46f5-a4be-94a63bc7a7e8 service nova] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:35.903271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165fec95-00f0-4c2a-afaf-1b5f96fbdec7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:35.903906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165fec95-00f0-4c2a-afaf-1b5f96fbdec7 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:35.904190 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165fec95-00f0-4c2a-afaf-1b5f96fbdec7 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:35.905100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165fec95-00f0-4c2a-afaf-1b5f96fbdec7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:35.907093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-165fec95-00f0-4c2a-afaf-1b5f96fbdec7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:35.978741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165fec95-00f0-4c2a-afaf-1b5f96fbdec7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:35.980916 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165fec95-00f0-4c2a-afaf-1b5f96fbdec7 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:35.991633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1153/2290] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:53:35 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:53:37.305586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3648f5-d5e1-44c0-8361-05025b2b382a None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:37.306096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3648f5-d5e1-44c0-8361-05025b2b382a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:37.306096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3648f5-d5e1-44c0-8361-05025b2b382a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:37.326736 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-0a3648f5-d5e1-44c0-8361-05025b2b382a None None] MFA Rules not processed for user `77b4b56f008c44c081130377cf8ce630`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:53:37.328343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0a3648f5-d5e1-44c0-8361-05025b2b382a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:37.387103 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1138/2291] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:53:37 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:53:37.802060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165fec95-00f0-4c2a-afaf-1b5f96fbdec7 service nova] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:37.803638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-165fec95-00f0-4c2a-afaf-1b5f96fbdec7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:37.866589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165fec95-00f0-4c2a-afaf-1b5f96fbdec7 service nova] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:37.870617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d125098e-8943-44be-81fd-8b9ea54cc167 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:37.871240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d125098e-8943-44be-81fd-8b9ea54cc167 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:37.871512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d125098e-8943-44be-81fd-8b9ea54cc167 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:37.871780 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d125098e-8943-44be-81fd-8b9ea54cc167 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:37.874167 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d125098e-8943-44be-81fd-8b9ea54cc167 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:37.885109 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d125098e-8943-44be-81fd-8b9ea54cc167 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:37.886576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d125098e-8943-44be-81fd-8b9ea54cc167 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:37.895636 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1154/2292] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:53:37 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:53:38.436775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3648f5-d5e1-44c0-8361-05025b2b382a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:38.439483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0a3648f5-d5e1-44c0-8361-05025b2b382a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:38.537339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3648f5-d5e1-44c0-8361-05025b2b382a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:38.543662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787afd3e-5aca-467d-a685-a5f43517ec86 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:38.544283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787afd3e-5aca-467d-a685-a5f43517ec86 service placement] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:38.544561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787afd3e-5aca-467d-a685-a5f43517ec86 service placement] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:38.544871 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787afd3e-5aca-467d-a685-a5f43517ec86 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:38.548159 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-787afd3e-5aca-467d-a685-a5f43517ec86 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:38.563658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787afd3e-5aca-467d-a685-a5f43517ec86 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:38.565958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787afd3e-5aca-467d-a685-a5f43517ec86 service placement] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:38.579778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1139/2293] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:53:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:53:45.885676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d125098e-8943-44be-81fd-8b9ea54cc167 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:53:45.887832 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d125098e-8943-44be-81fd-8b9ea54cc167 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:45.901087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d125098e-8943-44be-81fd-8b9ea54cc167 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:53:45.905753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8b04d7-03f0-4953-b46d-5ee1b637883b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:53:45.906691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8b04d7-03f0-4953-b46d-5ee1b637883b service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:45.906962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8b04d7-03f0-4953-b46d-5ee1b637883b service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:45.907236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8b04d7-03f0-4953-b46d-5ee1b637883b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:45.909010 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-dc8b04d7-03f0-4953-b46d-5ee1b637883b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:45.920867 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8b04d7-03f0-4953-b46d-5ee1b637883b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:53:45.922748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8b04d7-03f0-4953-b46d-5ee1b637883b service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:53:45.927467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:53:45.933893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1155/2294] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:53:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:53:48.572626 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=63,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:53:48.797787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338722d7-af91-48c0-bd8f-c7242b35d6f9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:53:48.798313 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338722d7-af91-48c0-bd8f-c7242b35d6f9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:53:48.798443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338722d7-af91-48c0-bd8f-c7242b35d6f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:53:48.819995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-338722d7-af91-48c0-bd8f-c7242b35d6f9 None None] MFA Rules not processed for user `0772f8b4306f4e239f075f7b1c13790f`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:53:48.821219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-338722d7-af91-48c0-bd8f-c7242b35d6f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:53:48.900939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1140/2295] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:53:48 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:53:55.961766 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:53:58.890750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=25 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:54:02.396874 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8b04d7-03f0-4953-b46d-5ee1b637883b service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:54:02.399055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-dc8b04d7-03f0-4953-b46d-5ee1b637883b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:54:02.429788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8b04d7-03f0-4953-b46d-5ee1b637883b service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:54:02.437687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a670a5bd-f2aa-467f-aabc-2b411feeae73 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:54:02.437899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a670a5bd-f2aa-467f-aabc-2b411feeae73 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:54:02.438058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a670a5bd-f2aa-467f-aabc-2b411feeae73 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:54:02.438360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a670a5bd-f2aa-467f-aabc-2b411feeae73 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:54:02.440232 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a670a5bd-f2aa-467f-aabc-2b411feeae73 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:54:02.453404 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a670a5bd-f2aa-467f-aabc-2b411feeae73 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:54:02.455103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a670a5bd-f2aa-467f-aabc-2b411feeae73 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:54:02.466753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1156/2296] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:54:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:54:09.714631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338722d7-af91-48c0-bd8f-c7242b35d6f9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:54:09.716169 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-338722d7-af91-48c0-bd8f-c7242b35d6f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:54:09.734339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338722d7-af91-48c0-bd8f-c7242b35d6f9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:54:09.740412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aaec54-f082-4b7b-86a5-6cfdb25e5614 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:54:09.741080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aaec54-f082-4b7b-86a5-6cfdb25e5614 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:54:09.741361 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aaec54-f082-4b7b-86a5-6cfdb25e5614 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:54:09.741791 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aaec54-f082-4b7b-86a5-6cfdb25e5614 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:54:09.743731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-52aaec54-f082-4b7b-86a5-6cfdb25e5614 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:54:09.757955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52aaec54-f082-4b7b-86a5-6cfdb25e5614 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:54:09.759675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52aaec54-f082-4b7b-86a5-6cfdb25e5614 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:54:09.772922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1141/2297] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:54:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:54:12.462561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=6 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:54:19.768303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=6 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:54:36.258604 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a670a5bd-f2aa-467f-aabc-2b411feeae73 service nova] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:54:36.269957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a670a5bd-f2aa-467f-aabc-2b411feeae73 service nova] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:54:36.275618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468c6699-4f01-4459-bd6b-6fba9b6ac237 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:54:36.276220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468c6699-4f01-4459-bd6b-6fba9b6ac237 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:54:36.276499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468c6699-4f01-4459-bd6b-6fba9b6ac237 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:54:36.276770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468c6699-4f01-4459-bd6b-6fba9b6ac237 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:54:36.278554 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-468c6699-4f01-4459-bd6b-6fba9b6ac237 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:54:36.337917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468c6699-4f01-4459-bd6b-6fba9b6ac237 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:54:36.339552 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468c6699-4f01-4459-bd6b-6fba9b6ac237 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:54:36.349305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1157/2298] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:54:36 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:54:46.344941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=28 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:54:50.835448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aaec54-f082-4b7b-86a5-6cfdb25e5614 service nova] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:54:50.837107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-52aaec54-f082-4b7b-86a5-6cfdb25e5614 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:54:50.914680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52aaec54-f082-4b7b-86a5-6cfdb25e5614 service nova] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:54:50.919638 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:54:50.920309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:54:50.920577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:54:50.920842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] PATH_INFO: `/v3/users/5247fd6c30794dc48f60a827d5837674` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:54:50.921995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] RBAC: Authorizing `identity:delete_user(user_id=5247fd6c30794dc48f60a827d5837674)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:54:50.924130 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:54:51.102431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5247fd6c30794dc48f60a827d5837674'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:54:51.103090 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5247fd6c30794dc48f60a827d5837674'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:54:51.110680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5247fd6c30794dc48f60a827d5837674'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:54:51.125405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5247fd6c30794dc48f60a827d5837674'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:54:51.236872 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5247fd6c30794dc48f60a827d5837674'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:54:51.243400 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1142/2299] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:54:50 2022] DELETE /identity/v3/users/5247fd6c30794dc48f60a827d5837674 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:54:51.431163 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f921b04-bcfb-4ffa-9d7c-0776cb1691f5 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:54:51.431706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f921b04-bcfb-4ffa-9d7c-0776cb1691f5 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:54:51.431837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f921b04-bcfb-4ffa-9d7c-0776cb1691f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:54:51.452757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-0f921b04-bcfb-4ffa-9d7c-0776cb1691f5 None None] MFA Rules not processed for user `bd8d967d9ecb410ea47e130ac3c8b4a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:54:51.453986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0f921b04-bcfb-4ffa-9d7c-0776cb1691f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:54:51.511439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1158/2300] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:54:51 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:54:51.551203 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:54:51.553599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:54:51.628774 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095e2485-f6fa-47bd-b5be-e813217a6f8f admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:54:51.633967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:54:51.634497 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:54:51.634776 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:54:51.635049 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69 admin admin] PATH_INFO: `/v3/projects/46f646fe435e4aa2954af01070537ae4` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:54:51.636182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69 admin admin] RBAC: Authorizing `identity:delete_project(project_id=46f646fe435e4aa2954af01070537ae4)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:54:51.638365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:54:51.661757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:54:51.767194 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '46f646fe435e4aa2954af01070537ae4', 'request_id': 'req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:54:51.774522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '46f646fe435e4aa2954af01070537ae4', 'request_id': 'req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:54:51.782840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '46f646fe435e4aa2954af01070537ae4', 'request_id': 'req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:54:51.850312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69 admin admin] The token cache is being invalidate because project 46f646fe435e4aa2954af01070537ae4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:54:51.850999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:54:51.851342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-35bcd050-5cf7-4a3e-b9da-a85c0a9c7e69 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:54:51.852975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1143/2301] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:54:51 2022] DELETE /identity/v3/projects/46f646fe435e4aa2954af01070537ae4 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:55:01.504673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=25 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:55:01.991246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:DELETE=7,keystone:SELECT=15,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:55:02.387017 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f921b04-bcfb-4ffa-9d7c-0776cb1691f5 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:02.387017 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0f921b04-bcfb-4ffa-9d7c-0776cb1691f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:02.465978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f921b04-bcfb-4ffa-9d7c-0776cb1691f5 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:02.530535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8828d0-aea1-4192-88b0-3016131028c9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:02.531173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8828d0-aea1-4192-88b0-3016131028c9 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:02.531339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8828d0-aea1-4192-88b0-3016131028c9 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:02.531613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8828d0-aea1-4192-88b0-3016131028c9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:02.533379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-da8828d0-aea1-4192-88b0-3016131028c9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:02.545650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8828d0-aea1-4192-88b0-3016131028c9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:02.549201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8828d0-aea1-4192-88b0-3016131028c9 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:02.561116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1159/2302] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:55:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:55:12.556019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=62 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:55:21.369260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1184a7ca-0880-4105-a6bd-76b425b5534a None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:21.369859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1184a7ca-0880-4105-a6bd-76b425b5534a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:21.369859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1184a7ca-0880-4105-a6bd-76b425b5534a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:21.392214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1184a7ca-0880-4105-a6bd-76b425b5534a None None] MFA Rules not processed for user `7f4991f9c6ca4fc99f756f75f9460808`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:55:21.393480 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1184a7ca-0880-4105-a6bd-76b425b5534a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:21.453189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1144/2303] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:55:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:55:21.779869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8828d0-aea1-4192-88b0-3016131028c9 service nova] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:21.781357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-da8828d0-aea1-4192-88b0-3016131028c9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:21.838237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8828d0-aea1-4192-88b0-3016131028c9 service nova] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:21.842859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7c51a4-da66-4dad-9cc2-929e905180d8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:21.843463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7c51a4-da66-4dad-9cc2-929e905180d8 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:21.843740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7c51a4-da66-4dad-9cc2-929e905180d8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:21.844011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7c51a4-da66-4dad-9cc2-929e905180d8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:21.845810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-fa7c51a4-da66-4dad-9cc2-929e905180d8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:21.855907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7c51a4-da66-4dad-9cc2-929e905180d8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:21.857500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7c51a4-da66-4dad-9cc2-929e905180d8 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:21.865932 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1160/2304] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:55:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:55:31.446302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=25 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:55:31.862096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=26 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:55:37.566977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1184a7ca-0880-4105-a6bd-76b425b5534a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:37.569137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1184a7ca-0880-4105-a6bd-76b425b5534a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:37.582104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1184a7ca-0880-4105-a6bd-76b425b5534a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:37.586832 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10749417-9e32-4bca-be0b-95b60a310f08 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:37.587672 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10749417-9e32-4bca-be0b-95b60a310f08 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:37.587963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10749417-9e32-4bca-be0b-95b60a310f08 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:37.588225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10749417-9e32-4bca-be0b-95b60a310f08 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:37.590319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-10749417-9e32-4bca-be0b-95b60a310f08 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:37.667108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10749417-9e32-4bca-be0b-95b60a310f08 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:37.669618 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10749417-9e32-4bca-be0b-95b60a310f08 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:37.680973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1145/2305] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:55:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:55:42.482891 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7c51a4-da66-4dad-9cc2-929e905180d8 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:42.488521 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-fa7c51a4-da66-4dad-9cc2-929e905180d8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:42.618064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7c51a4-da66-4dad-9cc2-929e905180d8 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:42.624191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526508c1-3f24-49a6-abb5-ef3d94a08b72 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:42.624816 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526508c1-3f24-49a6-abb5-ef3d94a08b72 service placement] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:42.625092 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526508c1-3f24-49a6-abb5-ef3d94a08b72 service placement] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:42.625362 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526508c1-3f24-49a6-abb5-ef3d94a08b72 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:42.627353 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-526508c1-3f24-49a6-abb5-ef3d94a08b72 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:42.641482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526508c1-3f24-49a6-abb5-ef3d94a08b72 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:42.643607 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526508c1-3f24-49a6-abb5-ef3d94a08b72 service placement] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:42.671285 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1161/2306] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:55:42 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:55:42.678894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10749417-9e32-4bca-be0b-95b60a310f08 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:42.680324 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-10749417-9e32-4bca-be0b-95b60a310f08 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:42.711438 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10749417-9e32-4bca-be0b-95b60a310f08 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:42.727790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a635c-8a93-4baf-be9a-5012cd9071e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:42.728824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a635c-8a93-4baf-be9a-5012cd9071e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:42.729248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a635c-8a93-4baf-be9a-5012cd9071e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:42.729726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a635c-8a93-4baf-be9a-5012cd9071e8 admin admin] PATH_INFO: `/v3/users/0772f8b4306f4e239f075f7b1c13790f` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:42.732978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a635c-8a93-4baf-be9a-5012cd9071e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0772f8b4306f4e239f075f7b1c13790f)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:42.745044 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a635c-8a93-4baf-be9a-5012cd9071e8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:43.019146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-cf7a635c-8a93-4baf-be9a-5012cd9071e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0772f8b4306f4e239f075f7b1c13790f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:43.020286 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-cf7a635c-8a93-4baf-be9a-5012cd9071e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0772f8b4306f4e239f075f7b1c13790f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:43.032268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-cf7a635c-8a93-4baf-be9a-5012cd9071e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0772f8b4306f4e239f075f7b1c13790f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:43.053424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-cf7a635c-8a93-4baf-be9a-5012cd9071e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0772f8b4306f4e239f075f7b1c13790f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:43.192521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-cf7a635c-8a93-4baf-be9a-5012cd9071e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0772f8b4306f4e239f075f7b1c13790f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:43.199644 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1146/2307] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:55:42 2022] DELETE /identity/v3/users/0772f8b4306f4e239f075f7b1c13790f => generated 0 bytes in 521 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:55:43.513988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526508c1-3f24-49a6-abb5-ef3d94a08b72 service placement] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:43.537806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526508c1-3f24-49a6-abb5-ef3d94a08b72 service placement] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:43.604540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc9554e-d3ff-45f7-919f-c6796111bad1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:43.605635 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc9554e-d3ff-45f7-919f-c6796111bad1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:43.606080 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc9554e-d3ff-45f7-919f-c6796111bad1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:43.606530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc9554e-d3ff-45f7-919f-c6796111bad1 admin admin] PATH_INFO: `/v3/projects/c993aed86aaf431495a2d1724218fa9b` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:43.607909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc9554e-d3ff-45f7-919f-c6796111bad1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c993aed86aaf431495a2d1724218fa9b)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:43.609975 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc9554e-d3ff-45f7-919f-c6796111bad1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:43.747237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3cc9554e-d3ff-45f7-919f-c6796111bad1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c993aed86aaf431495a2d1724218fa9b', 'request_id': 'req-3cc9554e-d3ff-45f7-919f-c6796111bad1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:43.754040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3cc9554e-d3ff-45f7-919f-c6796111bad1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c993aed86aaf431495a2d1724218fa9b', 'request_id': 'req-3cc9554e-d3ff-45f7-919f-c6796111bad1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:43.844192 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3cc9554e-d3ff-45f7-919f-c6796111bad1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c993aed86aaf431495a2d1724218fa9b', 'request_id': 'req-3cc9554e-d3ff-45f7-919f-c6796111bad1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:43.857115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3cc9554e-d3ff-45f7-919f-c6796111bad1 admin admin] The token cache is being invalidate because project c993aed86aaf431495a2d1724218fa9b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:55:43.857401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3cc9554e-d3ff-45f7-919f-c6796111bad1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:43.858069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3cc9554e-d3ff-45f7-919f-c6796111bad1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:43.859611 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1162/2308] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:55:43 2022] DELETE /identity/v3/projects/c993aed86aaf431495a2d1724218fa9b => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:55:43.865019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afed693-0e65-4bb6-b324-57cae9ebae48 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:43.865623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afed693-0e65-4bb6-b324-57cae9ebae48 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:43.865623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afed693-0e65-4bb6-b324-57cae9ebae48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:43.885721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-7afed693-0e65-4bb6-b324-57cae9ebae48 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:55:43.887865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7afed693-0e65-4bb6-b324-57cae9ebae48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:43.952160 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1147/2309] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:55:43.960069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbc6116-9418-48f2-b701-db7e8eea324b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:43.960355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbc6116-9418-48f2-b701-db7e8eea324b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:43.960631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbc6116-9418-48f2-b701-db7e8eea324b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:43.981923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-9cbc6116-9418-48f2-b701-db7e8eea324b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:55:43.983115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9cbc6116-9418-48f2-b701-db7e8eea324b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:43.991371 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1163/2310] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:55:43.997564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afed693-0e65-4bb6-b324-57cae9ebae48 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:44.007414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afed693-0e65-4bb6-b324-57cae9ebae48 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:44.013054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cbac35-9d0f-4bc4-802e-d77c855113e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:44.013661 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cbac35-9d0f-4bc4-802e-d77c855113e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:44.014111 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cbac35-9d0f-4bc4-802e-d77c855113e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:44.014416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cbac35-9d0f-4bc4-802e-d77c855113e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:44.015061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cbac35-9d0f-4bc4-802e-d77c855113e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:55:44.015377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cbac35-9d0f-4bc4-802e-d77c855113e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:44.016933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cbac35-9d0f-4bc4-802e-d77c855113e8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:44.018708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:55:44.030042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1148/2311] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:55:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:55:44.034349 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbc6116-9418-48f2-b701-db7e8eea324b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:44.045032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbc6116-9418-48f2-b701-db7e8eea324b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:44.053093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3efb8e-f330-42a3-b157-a323fc769f6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:44.053813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3efb8e-f330-42a3-b157-a323fc769f6e admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:44.054079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3efb8e-f330-42a3-b157-a323fc769f6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:44.054382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3efb8e-f330-42a3-b157-a323fc769f6e admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:44.055016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3efb8e-f330-42a3-b157-a323fc769f6e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:44.056978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3efb8e-f330-42a3-b157-a323fc769f6e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:44.071228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:55:44.163139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1164/2312] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:55:44 2022] POST /identity/v3/projects => generated 373 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:55:44.167308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cbac35-9d0f-4bc4-802e-d77c855113e8 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:44.177528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cbac35-9d0f-4bc4-802e-d77c855113e8 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:44.238685 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad4a1a3-bed2-4bf9-a974-2d835f0e2c41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:44.239274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad4a1a3-bed2-4bf9-a974-2d835f0e2c41 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:44.239552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad4a1a3-bed2-4bf9-a974-2d835f0e2c41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:44.239823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad4a1a3-bed2-4bf9-a974-2d835f0e2c41 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:44.240455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad4a1a3-bed2-4bf9-a974-2d835f0e2c41 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:44.242459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad4a1a3-bed2-4bf9-a974-2d835f0e2c41 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:44.330888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1149/2313] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:55:44 2022] POST /identity/v3/users => generated 287 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:55:44.335014 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3efb8e-f330-42a3-b157-a323fc769f6e admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:44.346258 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3efb8e-f330-42a3-b157-a323fc769f6e admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:44.353082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4beaeb-e419-4974-8bfa-3abae02812ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:44.354017 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4beaeb-e419-4974-8bfa-3abae02812ca admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:44.354395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4beaeb-e419-4974-8bfa-3abae02812ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:44.354741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4beaeb-e419-4974-8bfa-3abae02812ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:44.355852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4beaeb-e419-4974-8bfa-3abae02812ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:44.357652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4beaeb-e419-4974-8bfa-3abae02812ca admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:44.372442 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1165/2314] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:55:44 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:55:44.374478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad4a1a3-bed2-4bf9-a974-2d835f0e2c41 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:44.385671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad4a1a3-bed2-4bf9-a974-2d835f0e2c41 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:44.392379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9248459a-663f-4048-8d5e-367d5facac47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:44.392998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9248459a-663f-4048-8d5e-367d5facac47 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:44.396505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9248459a-663f-4048-8d5e-367d5facac47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:44.396505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9248459a-663f-4048-8d5e-367d5facac47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:44.396505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9248459a-663f-4048-8d5e-367d5facac47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:44.396505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9248459a-663f-4048-8d5e-367d5facac47 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:44.417444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1150/2315] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:55:44 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:55:44.422024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4beaeb-e419-4974-8bfa-3abae02812ca admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:44.436060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4beaeb-e419-4974-8bfa-3abae02812ca admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:44.440962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d877e1e0-3d95-493e-89bd-53b2d64b8452 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:44.441617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d877e1e0-3d95-493e-89bd-53b2d64b8452 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:44.441899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d877e1e0-3d95-493e-89bd-53b2d64b8452 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:44.442295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d877e1e0-3d95-493e-89bd-53b2d64b8452 admin admin] PATH_INFO: `/v3/projects/7704ebd24a8a4613bdfbdd1fbc3622c9/users/197934dd4e054f52af516b538fe9bd0a/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:44.458690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d877e1e0-3d95-493e-89bd-53b2d64b8452 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7704ebd24a8a4613bdfbdd1fbc3622c9, user_id=197934dd4e054f52af516b538fe9bd0a, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:44.461553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d877e1e0-3d95-493e-89bd-53b2d64b8452 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:44.578864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1166/2316] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:55:44 2022] PUT /identity/v3/projects/7704ebd24a8a4613bdfbdd1fbc3622c9/users/197934dd4e054f52af516b538fe9bd0a/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:55:44.588921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b527ee2e-0926-43db-8f1b-d5772f9f98d3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:44.589391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b527ee2e-0926-43db-8f1b-d5772f9f98d3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:44.590210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b527ee2e-0926-43db-8f1b-d5772f9f98d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:44.619576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-b527ee2e-0926-43db-8f1b-d5772f9f98d3 None None] MFA Rules not processed for user `197934dd4e054f52af516b538fe9bd0a`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:55:44.621061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b527ee2e-0926-43db-8f1b-d5772f9f98d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:44.691277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e18c63-aa14-4658-8be3-aa0cb9e2975d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:44.691645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e18c63-aa14-4658-8be3-aa0cb9e2975d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:44.691925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e18c63-aa14-4658-8be3-aa0cb9e2975d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:44.692448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1151/2317] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:55:44 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:55:44.704122 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b527ee2e-0926-43db-8f1b-d5772f9f98d3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:44.705566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b527ee2e-0926-43db-8f1b-d5772f9f98d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:44.712879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-81e18c63-aa14-4658-8be3-aa0cb9e2975d None None] MFA Rules not processed for user `77b4b56f008c44c081130377cf8ce630`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:55:44.714083 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-81e18c63-aa14-4658-8be3-aa0cb9e2975d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:44.777045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1167/2318] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:55:44 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:55:44.784370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b527ee2e-0926-43db-8f1b-d5772f9f98d3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:44.790126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656a35c8-934c-4c3a-b5fe-5b2063986236 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:44.790796 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656a35c8-934c-4c3a-b5fe-5b2063986236 service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:44.791070 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656a35c8-934c-4c3a-b5fe-5b2063986236 service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:44.791343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656a35c8-934c-4c3a-b5fe-5b2063986236 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:44.799886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656a35c8-934c-4c3a-b5fe-5b2063986236 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:44.801508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656a35c8-934c-4c3a-b5fe-5b2063986236 service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:44.811434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1152/2319] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:55:44 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:55:44.856366 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:55:48.261776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e18c63-aa14-4658-8be3-aa0cb9e2975d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:48.329688 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e18c63-aa14-4658-8be3-aa0cb9e2975d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:48.336353 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:48.337080 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:48.337369 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:48.337644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] PATH_INFO: `/v3/users/197934dd4e054f52af516b538fe9bd0a` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:48.339191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] RBAC: Authorizing `identity:delete_user(user_id=197934dd4e054f52af516b538fe9bd0a)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:48.349387 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:48.618895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '197934dd4e054f52af516b538fe9bd0a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:48.619643 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '197934dd4e054f52af516b538fe9bd0a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:48.628073 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '197934dd4e054f52af516b538fe9bd0a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:48.628747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '197934dd4e054f52af516b538fe9bd0a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:48.706873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '197934dd4e054f52af516b538fe9bd0a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:48.708539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:55:48.726267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1168/2320] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:55:48 2022] DELETE /identity/v3/users/197934dd4e054f52af516b538fe9bd0a => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:55:48.731539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4e5009-430d-4392-b1ad-79cf4c8da42d None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:48.732070 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4e5009-430d-4392-b1ad-79cf4c8da42d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:48.732307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4e5009-430d-4392-b1ad-79cf4c8da42d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:48.753377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-4c4e5009-430d-4392-b1ad-79cf4c8da42d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:55:48.754679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4c4e5009-430d-4392-b1ad-79cf4c8da42d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:48.834340 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1153/2321] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:55:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:55:48.839005 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:48.840839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:48.907024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cdc91c-263c-404e-9f48-d875e8364479 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:48.911998 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2209ec91-5496-412d-8297-d63f3d62bc05 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:48.911998 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2209ec91-5496-412d-8297-d63f3d62bc05 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:48.911998 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2209ec91-5496-412d-8297-d63f3d62bc05 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:48.912327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2209ec91-5496-412d-8297-d63f3d62bc05 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:48.922887 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2209ec91-5496-412d-8297-d63f3d62bc05 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:48.928682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2209ec91-5496-412d-8297-d63f3d62bc05 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:48.943912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1169/2322] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:55:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:55:49.428358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4e5009-430d-4392-b1ad-79cf4c8da42d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:49.430143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4c4e5009-430d-4392-b1ad-79cf4c8da42d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:49.444362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4e5009-430d-4392-b1ad-79cf4c8da42d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:49.449669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac9cc3-6ccd-40be-a837-9ce508531e50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:49.450521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac9cc3-6ccd-40be-a837-9ce508531e50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:49.450903 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac9cc3-6ccd-40be-a837-9ce508531e50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:49.451384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac9cc3-6ccd-40be-a837-9ce508531e50 admin admin] PATH_INFO: `/v3/projects/7704ebd24a8a4613bdfbdd1fbc3622c9` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:49.452854 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac9cc3-6ccd-40be-a837-9ce508531e50 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7704ebd24a8a4613bdfbdd1fbc3622c9)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:49.455010 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac9cc3-6ccd-40be-a837-9ce508531e50 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:49.562502 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-caac9cc3-6ccd-40be-a837-9ce508531e50 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7704ebd24a8a4613bdfbdd1fbc3622c9', 'request_id': 'req-caac9cc3-6ccd-40be-a837-9ce508531e50'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:49.569580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-caac9cc3-6ccd-40be-a837-9ce508531e50 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7704ebd24a8a4613bdfbdd1fbc3622c9', 'request_id': 'req-caac9cc3-6ccd-40be-a837-9ce508531e50'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:49.581198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-caac9cc3-6ccd-40be-a837-9ce508531e50 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7704ebd24a8a4613bdfbdd1fbc3622c9', 'request_id': 'req-caac9cc3-6ccd-40be-a837-9ce508531e50'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:49.670717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-caac9cc3-6ccd-40be-a837-9ce508531e50 admin admin] The token cache is being invalidate because project 7704ebd24a8a4613bdfbdd1fbc3622c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:55:49.670970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-caac9cc3-6ccd-40be-a837-9ce508531e50 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:49.671610 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-caac9cc3-6ccd-40be-a837-9ce508531e50 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:55:49.676734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1154/2323] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:55:49 2022] DELETE /identity/v3/projects/7704ebd24a8a4613bdfbdd1fbc3622c9 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:55:49.678379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d40e3c-a6ab-4185-b9fc-593a3df766f6 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:49.678960 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d40e3c-a6ab-4185-b9fc-593a3df766f6 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:49.679068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d40e3c-a6ab-4185-b9fc-593a3df766f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:49.700116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-45d40e3c-a6ab-4185-b9fc-593a3df766f6 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:55:49.701477 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-45d40e3c-a6ab-4185-b9fc-593a3df766f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:49.766781 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1170/2324] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:55:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:55:49.774757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1862cce-90de-42b9-90ab-197051685ded None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:49.775308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1862cce-90de-42b9-90ab-197051685ded None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:49.775463 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1862cce-90de-42b9-90ab-197051685ded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:49.798106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-c1862cce-90de-42b9-90ab-197051685ded None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:55:49.799359 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c1862cce-90de-42b9-90ab-197051685ded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:49.807612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1155/2325] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:55:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:55:49.812282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d40e3c-a6ab-4185-b9fc-593a3df766f6 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:49.822004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d40e3c-a6ab-4185-b9fc-593a3df766f6 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:49.827760 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2ae49c-e16f-4c1a-bd0e-4e2faf836341 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:49.828367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2ae49c-e16f-4c1a-bd0e-4e2faf836341 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:49.828640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2ae49c-e16f-4c1a-bd0e-4e2faf836341 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:49.828899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2ae49c-e16f-4c1a-bd0e-4e2faf836341 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:49.829552 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2ae49c-e16f-4c1a-bd0e-4e2faf836341 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:55:49.829845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2ae49c-e16f-4c1a-bd0e-4e2faf836341 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:49.831511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2ae49c-e16f-4c1a-bd0e-4e2faf836341 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:49.844822 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1171/2326] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:55:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:55:49.849349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1862cce-90de-42b9-90ab-197051685ded None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:49.859772 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1862cce-90de-42b9-90ab-197051685ded None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:49.864708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bdda7e-6066-4cff-bd2e-529ad42e6bfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:49.865330 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bdda7e-6066-4cff-bd2e-529ad42e6bfb admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:49.865581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bdda7e-6066-4cff-bd2e-529ad42e6bfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:49.865843 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bdda7e-6066-4cff-bd2e-529ad42e6bfb admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:49.866514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bdda7e-6066-4cff-bd2e-529ad42e6bfb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:49.870146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bdda7e-6066-4cff-bd2e-529ad42e6bfb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:49.959860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1156/2327] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:55:49 2022] POST /identity/v3/projects => generated 367 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:55:49.964333 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2ae49c-e16f-4c1a-bd0e-4e2faf836341 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:49.978275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2ae49c-e16f-4c1a-bd0e-4e2faf836341 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:50.033924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4560ac-26e4-472a-b5c2-2e00ac44def7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:50.034572 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4560ac-26e4-472a-b5c2-2e00ac44def7 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:50.034854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4560ac-26e4-472a-b5c2-2e00ac44def7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:50.035126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4560ac-26e4-472a-b5c2-2e00ac44def7 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:50.035871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4560ac-26e4-472a-b5c2-2e00ac44def7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:50.037871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4560ac-26e4-472a-b5c2-2e00ac44def7 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:50.165536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:55:50.176711 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1172/2328] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:55:49 2022] POST /identity/v3/users => generated 284 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:55:50.181242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bdda7e-6066-4cff-bd2e-529ad42e6bfb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:50.192742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bdda7e-6066-4cff-bd2e-529ad42e6bfb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:50.197467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e86099e-2d6d-4db9-9919-49f2214e6fe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:50.201779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e86099e-2d6d-4db9-9919-49f2214e6fe2 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:50.201779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e86099e-2d6d-4db9-9919-49f2214e6fe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:50.201779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e86099e-2d6d-4db9-9919-49f2214e6fe2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:50.201779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e86099e-2d6d-4db9-9919-49f2214e6fe2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:50.201779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e86099e-2d6d-4db9-9919-49f2214e6fe2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:50.212480 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1157/2329] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:55:50 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:55:50.216899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4560ac-26e4-472a-b5c2-2e00ac44def7 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:50.227238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4560ac-26e4-472a-b5c2-2e00ac44def7 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:50.232751 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b188be-5f01-4729-9df0-02d9f94e830b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:50.233441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b188be-5f01-4729-9df0-02d9f94e830b admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:50.233786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b188be-5f01-4729-9df0-02d9f94e830b admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:50.234126 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b188be-5f01-4729-9df0-02d9f94e830b admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:50.234910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b188be-5f01-4729-9df0-02d9f94e830b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:50.236592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b188be-5f01-4729-9df0-02d9f94e830b admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:50.251418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1173/2330] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:55:50 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:55:50.252656 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e86099e-2d6d-4db9-9919-49f2214e6fe2 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:50.264444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e86099e-2d6d-4db9-9919-49f2214e6fe2 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:50.269753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbf82c1-b19e-49ba-bfa1-afa02aef418c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:50.270446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbf82c1-b19e-49ba-bfa1-afa02aef418c admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:50.270752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbf82c1-b19e-49ba-bfa1-afa02aef418c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:50.271067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbf82c1-b19e-49ba-bfa1-afa02aef418c admin admin] PATH_INFO: `/v3/projects/281a48a7d9e246d8ae66fb322bf3fa57/users/3a4a9a1bfeef4d84832bfa7b5a7bc4bc/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:50.288266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbf82c1-b19e-49ba-bfa1-afa02aef418c admin admin] RBAC: Authorizing `identity:create_grant(project_id=281a48a7d9e246d8ae66fb322bf3fa57, user_id=3a4a9a1bfeef4d84832bfa7b5a7bc4bc, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:50.311456 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbf82c1-b19e-49ba-bfa1-afa02aef418c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:50.463371 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1158/2331] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:55:50 2022] PUT /identity/v3/projects/281a48a7d9e246d8ae66fb322bf3fa57/users/3a4a9a1bfeef4d84832bfa7b5a7bc4bc/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:55:50.474592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b8f30a-8a56-4af0-b1be-f55cb217eca5 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:50.474964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b8f30a-8a56-4af0-b1be-f55cb217eca5 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:50.475208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b8f30a-8a56-4af0-b1be-f55cb217eca5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:50.506169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d1b8f30a-8a56-4af0-b1be-f55cb217eca5 None None] MFA Rules not processed for user `3a4a9a1bfeef4d84832bfa7b5a7bc4bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:55:50.508347 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d1b8f30a-8a56-4af0-b1be-f55cb217eca5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:50.598068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1174/2332] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:55:50 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:55:50.611204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbf82c1-b19e-49ba-bfa1-afa02aef418c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:55:50.613086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6cbf82c1-b19e-49ba-bfa1-afa02aef418c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:55:50.623942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:55:50.678629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbf82c1-b19e-49ba-bfa1-afa02aef418c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:55:50.682591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7aa1a48-8ee2-46c3-addf-72aaf4605c52 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:55:50.683952 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7aa1a48-8ee2-46c3-addf-72aaf4605c52 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:55:50.684231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7aa1a48-8ee2-46c3-addf-72aaf4605c52 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:55:50.684638 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7aa1a48-8ee2-46c3-addf-72aaf4605c52 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:55:50.694016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7aa1a48-8ee2-46c3-addf-72aaf4605c52 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:55:50.695458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7aa1a48-8ee2-46c3-addf-72aaf4605c52 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:55:50.705168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1159/2333] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:55:50 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:56:00.592817 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=37 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:00.958124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=23 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:01.893403 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b8f30a-8a56-4af0-b1be-f55cb217eca5 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:02.029900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b8f30a-8a56-4af0-b1be-f55cb217eca5 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:02.036185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:02.036823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:02.037101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:02.037368 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] PATH_INFO: `/v3/users/3a4a9a1bfeef4d84832bfa7b5a7bc4bc` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:02.038542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a4a9a1bfeef4d84832bfa7b5a7bc4bc)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:02.040501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:02.222953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a4a9a1bfeef4d84832bfa7b5a7bc4bc'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:02.223937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a4a9a1bfeef4d84832bfa7b5a7bc4bc'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:02.231986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a4a9a1bfeef4d84832bfa7b5a7bc4bc'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:02.232694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a4a9a1bfeef4d84832bfa7b5a7bc4bc'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:02.316200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a4a9a1bfeef4d84832bfa7b5a7bc4bc'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:02.337420 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1175/2334] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:56:01 2022] DELETE /identity/v3/users/3a4a9a1bfeef4d84832bfa7b5a7bc4bc => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:02.342290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a7d141-3fed-4641-8610-0587703b4aea None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:02.342758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a7d141-3fed-4641-8610-0587703b4aea None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:02.342874 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a7d141-3fed-4641-8610-0587703b4aea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:02.413355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-78a7d141-3fed-4641-8610-0587703b4aea None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:02.414888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-78a7d141-3fed-4641-8610-0587703b4aea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:02.501000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1160/2335] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:56:02.511697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:02.513471 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:02.562769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:02.600080 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964b79d2-2fe3-44eb-bc02-431780050bed admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:02.606144 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee74f10a-d03c-4674-bfb0-0de9e78ce4eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:02.606788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee74f10a-d03c-4674-bfb0-0de9e78ce4eb service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:02.607072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee74f10a-d03c-4674-bfb0-0de9e78ce4eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:02.607368 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee74f10a-d03c-4674-bfb0-0de9e78ce4eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:02.628969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee74f10a-d03c-4674-bfb0-0de9e78ce4eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:02.630574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee74f10a-d03c-4674-bfb0-0de9e78ce4eb service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:02.641287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1176/2336] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:56:03.405939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a7d141-3fed-4641-8610-0587703b4aea None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:03.407965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-78a7d141-3fed-4641-8610-0587703b4aea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:03.421745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a7d141-3fed-4641-8610-0587703b4aea None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:03.427672 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:03.428557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:03.428901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:03.429242 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9 admin admin] PATH_INFO: `/v3/projects/281a48a7d9e246d8ae66fb322bf3fa57` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:03.430809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=281a48a7d9e246d8ae66fb322bf3fa57)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:03.435041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:03.599678 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '281a48a7d9e246d8ae66fb322bf3fa57', 'request_id': 'req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:03.605967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '281a48a7d9e246d8ae66fb322bf3fa57', 'request_id': 'req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:03.612217 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '281a48a7d9e246d8ae66fb322bf3fa57', 'request_id': 'req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:03.741417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9 admin admin] The token cache is being invalidate because project 281a48a7d9e246d8ae66fb322bf3fa57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:56:03.741713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:03.742442 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-00c12c9f-ed6e-4cbc-a5a4-afe47ba2c4e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:03.744690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1161/2337] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:56:03 2022] DELETE /identity/v3/projects/281a48a7d9e246d8ae66fb322bf3fa57 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:03.749848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fcb728-a61c-4e10-813b-9626896c4f9d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:03.750466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fcb728-a61c-4e10-813b-9626896c4f9d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:03.750466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fcb728-a61c-4e10-813b-9626896c4f9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:03.806541 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-09fcb728-a61c-4e10-813b-9626896c4f9d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:03.808146 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-09fcb728-a61c-4e10-813b-9626896c4f9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:03.902254 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1177/2338] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:03.910675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3428f141-b336-4013-b6f8-62395fa4f735 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:03.911241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3428f141-b336-4013-b6f8-62395fa4f735 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:03.916012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3428f141-b336-4013-b6f8-62395fa4f735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:03.935073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-3428f141-b336-4013-b6f8-62395fa4f735 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:03.936413 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3428f141-b336-4013-b6f8-62395fa4f735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:03.945240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1162/2339] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:03.951566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fcb728-a61c-4e10-813b-9626896c4f9d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:04.152040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fcb728-a61c-4e10-813b-9626896c4f9d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:04.152040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308b94ff-f27f-4746-82d2-dc1ca1239739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:04.152040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308b94ff-f27f-4746-82d2-dc1ca1239739 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:04.152040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308b94ff-f27f-4746-82d2-dc1ca1239739 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:04.153132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308b94ff-f27f-4746-82d2-dc1ca1239739 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:04.153132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308b94ff-f27f-4746-82d2-dc1ca1239739 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:56:04.153132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308b94ff-f27f-4746-82d2-dc1ca1239739 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:04.153132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308b94ff-f27f-4746-82d2-dc1ca1239739 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:04.153132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1178/2340] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:56:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:56:04.153562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3428f141-b336-4013-b6f8-62395fa4f735 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:04.153562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3428f141-b336-4013-b6f8-62395fa4f735 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:04.153562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414889d9-1be4-41b3-af4c-14e910cfe0e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:04.153562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414889d9-1be4-41b3-af4c-14e910cfe0e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:04.153930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414889d9-1be4-41b3-af4c-14e910cfe0e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:04.153930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414889d9-1be4-41b3-af4c-14e910cfe0e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:04.153930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414889d9-1be4-41b3-af4c-14e910cfe0e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:04.153930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414889d9-1be4-41b3-af4c-14e910cfe0e9 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:04.244530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1163/2341] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:56:04 2022] POST /identity/v3/projects => generated 373 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:56:04.249552 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308b94ff-f27f-4746-82d2-dc1ca1239739 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:04.264294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308b94ff-f27f-4746-82d2-dc1ca1239739 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:04.343761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ab0523-bcec-4659-a8ab-cee92a38abef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:04.349450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ab0523-bcec-4659-a8ab-cee92a38abef admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:04.349450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ab0523-bcec-4659-a8ab-cee92a38abef admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:04.349450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ab0523-bcec-4659-a8ab-cee92a38abef admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:04.349450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ab0523-bcec-4659-a8ab-cee92a38abef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:04.349450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ab0523-bcec-4659-a8ab-cee92a38abef admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:04.459233 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1179/2342] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:56:04 2022] POST /identity/v3/users => generated 287 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:04.464816 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414889d9-1be4-41b3-af4c-14e910cfe0e9 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:04.475259 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414889d9-1be4-41b3-af4c-14e910cfe0e9 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:04.480485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e491b3f9-9c19-44d9-9024-7a3016cb9bc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:04.481105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e491b3f9-9c19-44d9-9024-7a3016cb9bc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:04.481387 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e491b3f9-9c19-44d9-9024-7a3016cb9bc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:04.481730 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e491b3f9-9c19-44d9-9024-7a3016cb9bc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:04.482405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e491b3f9-9c19-44d9-9024-7a3016cb9bc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:04.483998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e491b3f9-9c19-44d9-9024-7a3016cb9bc1 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:04.497702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1164/2343] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:04.498983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ab0523-bcec-4659-a8ab-cee92a38abef admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:04.508826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ab0523-bcec-4659-a8ab-cee92a38abef admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:04.515988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237e6b9e-b905-4c72-9c2c-442d63f98b36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:04.516821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237e6b9e-b905-4c72-9c2c-442d63f98b36 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:04.517101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237e6b9e-b905-4c72-9c2c-442d63f98b36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:04.517538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237e6b9e-b905-4c72-9c2c-442d63f98b36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:04.518618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237e6b9e-b905-4c72-9c2c-442d63f98b36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:04.520269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237e6b9e-b905-4c72-9c2c-442d63f98b36 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:04.535247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1180/2344] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:04 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:04.536558 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e491b3f9-9c19-44d9-9024-7a3016cb9bc1 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:04.549481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e491b3f9-9c19-44d9-9024-7a3016cb9bc1 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:04.554931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57146d00-31bb-49ec-9000-18b7336cac74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:04.560054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57146d00-31bb-49ec-9000-18b7336cac74 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:04.560054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57146d00-31bb-49ec-9000-18b7336cac74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:04.560054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57146d00-31bb-49ec-9000-18b7336cac74 admin admin] PATH_INFO: `/v3/projects/c53eb7ecb428418ca61ef5e5dc6252da/users/8ada345d494840f0ae873af5db1f882d/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:04.572306 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57146d00-31bb-49ec-9000-18b7336cac74 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c53eb7ecb428418ca61ef5e5dc6252da, user_id=8ada345d494840f0ae873af5db1f882d, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:04.575098 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57146d00-31bb-49ec-9000-18b7336cac74 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:04.577466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:04.697523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1165/2345] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:56:04 2022] PUT /identity/v3/projects/c53eb7ecb428418ca61ef5e5dc6252da/users/8ada345d494840f0ae873af5db1f882d/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:04.708291 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fa85a8-4f66-4fbb-95ed-d7fb3ad35986 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:04.708699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fa85a8-4f66-4fbb-95ed-d7fb3ad35986 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:04.708952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fa85a8-4f66-4fbb-95ed-d7fb3ad35986 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:04.727776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:04.750762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-37fa85a8-4f66-4fbb-95ed-d7fb3ad35986 None None] MFA Rules not processed for user `8ada345d494840f0ae873af5db1f882d`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:04.752330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-37fa85a8-4f66-4fbb-95ed-d7fb3ad35986 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:04.829775 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1181/2346] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:04 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:04.840965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57146d00-31bb-49ec-9000-18b7336cac74 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:04.842866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-57146d00-31bb-49ec-9000-18b7336cac74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:04.982009 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57146d00-31bb-49ec-9000-18b7336cac74 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:04.987778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5391c2-703d-45a9-b47d-5729cac636ac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:04.988417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5391c2-703d-45a9-b47d-5729cac636ac service nova] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:04.988660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5391c2-703d-45a9-b47d-5729cac636ac service nova] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:04.988928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5391c2-703d-45a9-b47d-5729cac636ac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:04.997750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5391c2-703d-45a9-b47d-5729cac636ac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:05.001381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5391c2-703d-45a9-b47d-5729cac636ac service nova] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:05.008925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1166/2347] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:04 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:56:05.226908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fa85a8-4f66-4fbb-95ed-d7fb3ad35986 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:05.293618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fa85a8-4f66-4fbb-95ed-d7fb3ad35986 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:05.298397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:05.299036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:05.299339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:05.299612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] PATH_INFO: `/v3/users/8ada345d494840f0ae873af5db1f882d` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:05.300947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ada345d494840f0ae873af5db1f882d)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:05.302942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:05.536713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ada345d494840f0ae873af5db1f882d'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:05.537432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ada345d494840f0ae873af5db1f882d'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:05.546143 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ada345d494840f0ae873af5db1f882d'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:05.547522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ada345d494840f0ae873af5db1f882d'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:05.630941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ada345d494840f0ae873af5db1f882d'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:05.667025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1182/2348] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:56:05 2022] DELETE /identity/v3/users/8ada345d494840f0ae873af5db1f882d => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:05.674427 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d50d854-3d3e-49ab-a2ca-3cf085ab0404 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:05.675313 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d50d854-3d3e-49ab-a2ca-3cf085ab0404 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:05.675601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d50d854-3d3e-49ab-a2ca-3cf085ab0404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:05.697566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-6d50d854-3d3e-49ab-a2ca-3cf085ab0404 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:05.701993 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6d50d854-3d3e-49ab-a2ca-3cf085ab0404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:05.777046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1167/2349] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:05.789533 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:05.796286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:05.819339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:05.866458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e2adf3-55b1-42d5-83b8-fbd34803f647 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:05.871076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de671380-2f45-448d-b237-c1fbf6428b4a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:05.871711 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de671380-2f45-448d-b237-c1fbf6428b4a service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:05.872007 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de671380-2f45-448d-b237-c1fbf6428b4a service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:05.872272 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de671380-2f45-448d-b237-c1fbf6428b4a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:05.882561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de671380-2f45-448d-b237-c1fbf6428b4a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:05.884009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de671380-2f45-448d-b237-c1fbf6428b4a service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:05.896740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1183/2350] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:56:05.936091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20abaeec-1582-4fa6-9118-86910ed9e262 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:05.936392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20abaeec-1582-4fa6-9118-86910ed9e262 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:05.936650 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20abaeec-1582-4fa6-9118-86910ed9e262 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:05.957224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-20abaeec-1582-4fa6-9118-86910ed9e262 None None] MFA Rules not processed for user `71ff73cade8b42c7ae57be1e3ef0e028`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:05.958496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-20abaeec-1582-4fa6-9118-86910ed9e262 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:06.011564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:06.020094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1168/2351] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:05 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:06.507728 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de671380-2f45-448d-b237-c1fbf6428b4a service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:06.509518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-de671380-2f45-448d-b237-c1fbf6428b4a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:06.523763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de671380-2f45-448d-b237-c1fbf6428b4a service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:06.529721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3dcaaf-bd0d-440b-b198-3837a0325243 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:06.530474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3dcaaf-bd0d-440b-b198-3837a0325243 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:06.530756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3dcaaf-bd0d-440b-b198-3837a0325243 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:06.531033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3dcaaf-bd0d-440b-b198-3837a0325243 admin admin] PATH_INFO: `/v3/projects/c53eb7ecb428418ca61ef5e5dc6252da` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:06.532326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3dcaaf-bd0d-440b-b198-3837a0325243 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c53eb7ecb428418ca61ef5e5dc6252da)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:06.534337 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3dcaaf-bd0d-440b-b198-3837a0325243 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:06.838093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4c3dcaaf-bd0d-440b-b198-3837a0325243 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c53eb7ecb428418ca61ef5e5dc6252da', 'request_id': 'req-4c3dcaaf-bd0d-440b-b198-3837a0325243'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:06.845240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4c3dcaaf-bd0d-440b-b198-3837a0325243 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c53eb7ecb428418ca61ef5e5dc6252da', 'request_id': 'req-4c3dcaaf-bd0d-440b-b198-3837a0325243'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:06.957542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4c3dcaaf-bd0d-440b-b198-3837a0325243 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c53eb7ecb428418ca61ef5e5dc6252da', 'request_id': 'req-4c3dcaaf-bd0d-440b-b198-3837a0325243'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:06.970430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4c3dcaaf-bd0d-440b-b198-3837a0325243 admin admin] The token cache is being invalidate because project c53eb7ecb428418ca61ef5e5dc6252da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:56:06.970722 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4c3dcaaf-bd0d-440b-b198-3837a0325243 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:06.971426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-4c3dcaaf-bd0d-440b-b198-3837a0325243 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:06.972939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1184/2352] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:56:06 2022] DELETE /identity/v3/projects/c53eb7ecb428418ca61ef5e5dc6252da => generated 0 bytes in 466 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:56:06.977826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecb7f1f-61a5-4c5c-af22-bca322652ac3 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:06.978345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecb7f1f-61a5-4c5c-af22-bca322652ac3 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:06.978453 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecb7f1f-61a5-4c5c-af22-bca322652ac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:07.000586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-cecb7f1f-61a5-4c5c-af22-bca322652ac3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:07.001841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-cecb7f1f-61a5-4c5c-af22-bca322652ac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:07.063283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1169/2353] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:07.072166 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3256c99-581b-4c9e-a35f-0b406d09709e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:07.072448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3256c99-581b-4c9e-a35f-0b406d09709e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:07.072720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3256c99-581b-4c9e-a35f-0b406d09709e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:07.098208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d3256c99-581b-4c9e-a35f-0b406d09709e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:07.098208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d3256c99-581b-4c9e-a35f-0b406d09709e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:07.102555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1185/2354] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:07.107914 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecb7f1f-61a5-4c5c-af22-bca322652ac3 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:07.118471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecb7f1f-61a5-4c5c-af22-bca322652ac3 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:07.124366 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177cfc5f-d541-4849-a9de-6ffe4fef6651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:07.126341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177cfc5f-d541-4849-a9de-6ffe4fef6651 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:07.126341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177cfc5f-d541-4849-a9de-6ffe4fef6651 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:07.126341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177cfc5f-d541-4849-a9de-6ffe4fef6651 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:07.126341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177cfc5f-d541-4849-a9de-6ffe4fef6651 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:56:07.126716 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177cfc5f-d541-4849-a9de-6ffe4fef6651 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:07.128151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177cfc5f-d541-4849-a9de-6ffe4fef6651 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:07.140765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1170/2355] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:56:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:56:07.145650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3256c99-581b-4c9e-a35f-0b406d09709e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:07.156004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3256c99-581b-4c9e-a35f-0b406d09709e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:07.160924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9824293-b695-4f16-864a-edb4bdefed95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:07.161615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9824293-b695-4f16-864a-edb4bdefed95 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:07.161979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9824293-b695-4f16-864a-edb4bdefed95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:07.162293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9824293-b695-4f16-864a-edb4bdefed95 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:07.162945 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9824293-b695-4f16-864a-edb4bdefed95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:07.164870 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9824293-b695-4f16-864a-edb4bdefed95 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:07.341044 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1186/2356] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:56:07 2022] POST /identity/v3/projects => generated 365 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:56:07.345540 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177cfc5f-d541-4849-a9de-6ffe4fef6651 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:07.357135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177cfc5f-d541-4849-a9de-6ffe4fef6651 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:07.430093 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773b6c09-65fa-4c78-b4e9-71f639ad9d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:07.431030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773b6c09-65fa-4c78-b4e9-71f639ad9d25 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:07.431030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773b6c09-65fa-4c78-b4e9-71f639ad9d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:07.431369 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773b6c09-65fa-4c78-b4e9-71f639ad9d25 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:07.431947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773b6c09-65fa-4c78-b4e9-71f639ad9d25 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:07.434203 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773b6c09-65fa-4c78-b4e9-71f639ad9d25 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:07.549693 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1171/2357] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:56:07 2022] POST /identity/v3/users => generated 283 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:07.554940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9824293-b695-4f16-864a-edb4bdefed95 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:07.565924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9824293-b695-4f16-864a-edb4bdefed95 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:07.570968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e2097a-4d25-4fb8-b127-57ca029fef7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:07.571576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e2097a-4d25-4fb8-b127-57ca029fef7e admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:07.571851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e2097a-4d25-4fb8-b127-57ca029fef7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:07.572122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e2097a-4d25-4fb8-b127-57ca029fef7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:07.573617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e2097a-4d25-4fb8-b127-57ca029fef7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:07.575525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e2097a-4d25-4fb8-b127-57ca029fef7e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:07.586575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1187/2358] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:07 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:07.591574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773b6c09-65fa-4c78-b4e9-71f639ad9d25 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:07.603282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773b6c09-65fa-4c78-b4e9-71f639ad9d25 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:07.608795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42efe4ff-7053-4edf-aaa7-9fda06e82232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:07.609340 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42efe4ff-7053-4edf-aaa7-9fda06e82232 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:07.609646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42efe4ff-7053-4edf-aaa7-9fda06e82232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:07.609917 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42efe4ff-7053-4edf-aaa7-9fda06e82232 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:07.610607 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42efe4ff-7053-4edf-aaa7-9fda06e82232 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:07.612239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42efe4ff-7053-4edf-aaa7-9fda06e82232 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:07.623973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1172/2359] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:07 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:07.628702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e2097a-4d25-4fb8-b127-57ca029fef7e admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:07.640347 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e2097a-4d25-4fb8-b127-57ca029fef7e admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:07.645247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6108e713-7a19-48d1-83f6-0a683c4e4e16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:07.645923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6108e713-7a19-48d1-83f6-0a683c4e4e16 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:07.646226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6108e713-7a19-48d1-83f6-0a683c4e4e16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:07.646496 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6108e713-7a19-48d1-83f6-0a683c4e4e16 admin admin] PATH_INFO: `/v3/projects/7436337f4e63421099bcdd05ef755557/users/f2bf6f4e36c04ea4b7945465b138b2ba/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:07.663970 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6108e713-7a19-48d1-83f6-0a683c4e4e16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7436337f4e63421099bcdd05ef755557, user_id=f2bf6f4e36c04ea4b7945465b138b2ba, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:07.666705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6108e713-7a19-48d1-83f6-0a683c4e4e16 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:07.775366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1188/2360] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:56:07 2022] PUT /identity/v3/projects/7436337f4e63421099bcdd05ef755557/users/f2bf6f4e36c04ea4b7945465b138b2ba/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:56:07.791231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75411c8d-8245-47eb-9387-37c1fb8772a7 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:07.791524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75411c8d-8245-47eb-9387-37c1fb8772a7 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:07.792294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75411c8d-8245-47eb-9387-37c1fb8772a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:07.825028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-75411c8d-8245-47eb-9387-37c1fb8772a7 None None] MFA Rules not processed for user `f2bf6f4e36c04ea4b7945465b138b2ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:07.827171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-75411c8d-8245-47eb-9387-37c1fb8772a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:07.927830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:07.939792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1173/2361] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:07 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:07.944583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6108e713-7a19-48d1-83f6-0a683c4e4e16 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:07.997741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:08.020606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6108e713-7a19-48d1-83f6-0a683c4e4e16 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:08.025544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eef3cec-d2f1-4c28-873d-0e20929b4c1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:08.026636 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eef3cec-d2f1-4c28-873d-0e20929b4c1b admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:08.026991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eef3cec-d2f1-4c28-873d-0e20929b4c1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:08.027276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eef3cec-d2f1-4c28-873d-0e20929b4c1b admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:08.027951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eef3cec-d2f1-4c28-873d-0e20929b4c1b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:08.029877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eef3cec-d2f1-4c28-873d-0e20929b4c1b admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:08.137250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1189/2362] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:56:07 2022] POST /identity/v3/projects => generated 369 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:08.142097 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75411c8d-8245-47eb-9387-37c1fb8772a7 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:08.155249 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75411c8d-8245-47eb-9387-37c1fb8772a7 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:08.229416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8eda69-18ae-4009-a7bb-943adb3dcdae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:08.230341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8eda69-18ae-4009-a7bb-943adb3dcdae admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:08.230621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8eda69-18ae-4009-a7bb-943adb3dcdae admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:08.230893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8eda69-18ae-4009-a7bb-943adb3dcdae admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:08.231532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8eda69-18ae-4009-a7bb-943adb3dcdae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:08.233503 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8eda69-18ae-4009-a7bb-943adb3dcdae admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:08.370707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1174/2363] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:56:08 2022] POST /identity/v3/users => generated 291 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:08.375177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eef3cec-d2f1-4c28-873d-0e20929b4c1b admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:08.387784 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eef3cec-d2f1-4c28-873d-0e20929b4c1b admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:08.392632 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1da946-d83e-4994-97c2-cd7ac0a2585f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:08.393269 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1da946-d83e-4994-97c2-cd7ac0a2585f admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:08.393577 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1da946-d83e-4994-97c2-cd7ac0a2585f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:08.393852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1da946-d83e-4994-97c2-cd7ac0a2585f admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:08.394553 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1da946-d83e-4994-97c2-cd7ac0a2585f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:08.396251 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1da946-d83e-4994-97c2-cd7ac0a2585f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:08.407797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1190/2364] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:08 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:08.415970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8eda69-18ae-4009-a7bb-943adb3dcdae admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:08.426128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8eda69-18ae-4009-a7bb-943adb3dcdae admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:08.433006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41990843-956b-4ed3-8385-6f18825813c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:08.433698 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41990843-956b-4ed3-8385-6f18825813c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:08.435541 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41990843-956b-4ed3-8385-6f18825813c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:08.435831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41990843-956b-4ed3-8385-6f18825813c9 admin admin] PATH_INFO: `/v3/projects/a66d56505d08471894c9ca2b1bd179a3/users/cfb448c9b5a140b189699fbf99529a48/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:08.453700 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41990843-956b-4ed3-8385-6f18825813c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a66d56505d08471894c9ca2b1bd179a3, user_id=cfb448c9b5a140b189699fbf99529a48, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:08.456421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41990843-956b-4ed3-8385-6f18825813c9 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:08.565104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1175/2365] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:56:08 2022] PUT /identity/v3/projects/a66d56505d08471894c9ca2b1bd179a3/users/cfb448c9b5a140b189699fbf99529a48/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:08.569397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1da946-d83e-4994-97c2-cd7ac0a2585f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:08.580613 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1da946-d83e-4994-97c2-cd7ac0a2585f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:08.642644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a90378-2541-4fa8-b6ba-bac3f1a51a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:08.643172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a90378-2541-4fa8-b6ba-bac3f1a51a63 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:08.643475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a90378-2541-4fa8-b6ba-bac3f1a51a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:08.643726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a90378-2541-4fa8-b6ba-bac3f1a51a63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:08.644387 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a90378-2541-4fa8-b6ba-bac3f1a51a63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:08.646098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a90378-2541-4fa8-b6ba-bac3f1a51a63 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:08.657893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1191/2366] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:08 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:56:08.662446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41990843-956b-4ed3-8385-6f18825813c9 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:08.674106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41990843-956b-4ed3-8385-6f18825813c9 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:08.682277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe823d1-edef-41d4-b146-abde460e78c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:08.682277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe823d1-edef-41d4-b146-abde460e78c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:08.682277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe823d1-edef-41d4-b146-abde460e78c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:08.682277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe823d1-edef-41d4-b146-abde460e78c6 admin admin] PATH_INFO: `/v3/projects/a66d56505d08471894c9ca2b1bd179a3/users/cfb448c9b5a140b189699fbf99529a48/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:08.682852 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe823d1-edef-41d4-b146-abde460e78c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a66d56505d08471894c9ca2b1bd179a3, user_id=cfb448c9b5a140b189699fbf99529a48, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:08.685019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe823d1-edef-41d4-b146-abde460e78c6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:08.697840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1176/2367] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:56:08 2022] PUT /identity/v3/projects/a66d56505d08471894c9ca2b1bd179a3/users/cfb448c9b5a140b189699fbf99529a48/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:08.709744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bf9ab3-e955-44dd-905e-7db851b793d8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:08.720921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bf9ab3-e955-44dd-905e-7db851b793d8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:08.720921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bf9ab3-e955-44dd-905e-7db851b793d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:08.741161 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c9bf9ab3-e955-44dd-905e-7db851b793d8 None None] MFA Rules not processed for user `cfb448c9b5a140b189699fbf99529a48`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:08.742422 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c9bf9ab3-e955-44dd-905e-7db851b793d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:08.813487 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1192/2368] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:08 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:08.825322 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe823d1-edef-41d4-b146-abde460e78c6 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:08.827201 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5fe823d1-edef-41d4-b146-abde460e78c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:08.894621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe823d1-edef-41d4-b146-abde460e78c6 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:08.898929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a88e94-23ce-4723-aef5-3bb9e977d4e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:08.899552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a88e94-23ce-4723-aef5-3bb9e977d4e8 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:08.899826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a88e94-23ce-4723-aef5-3bb9e977d4e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:08.900094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a88e94-23ce-4723-aef5-3bb9e977d4e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:08.910809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a88e94-23ce-4723-aef5-3bb9e977d4e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:08.912297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a88e94-23ce-4723-aef5-3bb9e977d4e8 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:08.924189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1177/2369] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:08 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:56:12.525326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bf9ab3-e955-44dd-905e-7db851b793d8 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:12.588107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:12.619929 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bf9ab3-e955-44dd-905e-7db851b793d8 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:12.625033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:12.625665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:12.625942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:12.626247 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] PATH_INFO: `/v3/users/f2bf6f4e36c04ea4b7945465b138b2ba` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:12.627384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2bf6f4e36c04ea4b7945465b138b2ba)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:12.629307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:13.047948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2bf6f4e36c04ea4b7945465b138b2ba'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:13.048635 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2bf6f4e36c04ea4b7945465b138b2ba'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:13.055569 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2bf6f4e36c04ea4b7945465b138b2ba'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:13.056259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2bf6f4e36c04ea4b7945465b138b2ba'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:13.169399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2bf6f4e36c04ea4b7945465b138b2ba'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:13.193666 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1193/2370] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:56:12 2022] DELETE /identity/v3/users/f2bf6f4e36c04ea4b7945465b138b2ba => generated 0 bytes in 669 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:13.200459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd05cd1-0568-4d8a-b56e-b3a93d59369a None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:13.201254 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd05cd1-0568-4d8a-b56e-b3a93d59369a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:13.201756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd05cd1-0568-4d8a-b56e-b3a93d59369a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:13.221665 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-1dd05cd1-0568-4d8a-b56e-b3a93d59369a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:13.222919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1dd05cd1-0568-4d8a-b56e-b3a93d59369a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:13.269485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:13.285436 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1178/2371] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:13.296446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:13.298224 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:13.362410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1489aa-e0c5-4d82-a227-475d02ed6a98 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:13.366135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff05bd7-a78f-40a3-9f44-8d03307ef98a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:13.366826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff05bd7-a78f-40a3-9f44-8d03307ef98a service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:13.367102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff05bd7-a78f-40a3-9f44-8d03307ef98a service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:13.367422 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff05bd7-a78f-40a3-9f44-8d03307ef98a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:13.377479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff05bd7-a78f-40a3-9f44-8d03307ef98a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:13.378955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff05bd7-a78f-40a3-9f44-8d03307ef98a service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:13.389460 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1194/2372] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:56:13.983931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd05cd1-0568-4d8a-b56e-b3a93d59369a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:13.987859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1dd05cd1-0568-4d8a-b56e-b3a93d59369a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:13.999669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd05cd1-0568-4d8a-b56e-b3a93d59369a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:14.006280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:14.006929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:14.012320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:14.012320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] PATH_INFO: `/v3/projects/7436337f4e63421099bcdd05ef755557` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:14.012320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7436337f4e63421099bcdd05ef755557)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:14.013321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:14.135925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7436337f4e63421099bcdd05ef755557', 'request_id': 'req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:14.141676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7436337f4e63421099bcdd05ef755557', 'request_id': 'req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:14.148635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7436337f4e63421099bcdd05ef755557', 'request_id': 'req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:14.252705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] The token cache is being invalidate because project 7436337f4e63421099bcdd05ef755557 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:56:14.253494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:14.254000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:14.255612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1179/2373] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:56:13 2022] DELETE /identity/v3/projects/7436337f4e63421099bcdd05ef755557 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:14.259747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff05bd7-a78f-40a3-9f44-8d03307ef98a service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:14.270022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff05bd7-a78f-40a3-9f44-8d03307ef98a service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:14.332923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:14.341162 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2f6c47-9120-4b39-9e6a-fd760c4279c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:14.341886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2f6c47-9120-4b39-9e6a-fd760c4279c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:14.342182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2f6c47-9120-4b39-9e6a-fd760c4279c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:14.342462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2f6c47-9120-4b39-9e6a-fd760c4279c8 admin admin] PATH_INFO: `/v3/users/cfb448c9b5a140b189699fbf99529a48` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:14.343935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2f6c47-9120-4b39-9e6a-fd760c4279c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cfb448c9b5a140b189699fbf99529a48)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:14.346245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2f6c47-9120-4b39-9e6a-fd760c4279c8 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:14.573314 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2d2f6c47-9120-4b39-9e6a-fd760c4279c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cfb448c9b5a140b189699fbf99529a48'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:14.574250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2d2f6c47-9120-4b39-9e6a-fd760c4279c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cfb448c9b5a140b189699fbf99529a48'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:14.580981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2d2f6c47-9120-4b39-9e6a-fd760c4279c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cfb448c9b5a140b189699fbf99529a48'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:14.581610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2d2f6c47-9120-4b39-9e6a-fd760c4279c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cfb448c9b5a140b189699fbf99529a48'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:14.677167 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-2d2f6c47-9120-4b39-9e6a-fd760c4279c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cfb448c9b5a140b189699fbf99529a48'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:14.696806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1195/2374] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:56:14 2022] DELETE /identity/v3/users/cfb448c9b5a140b189699fbf99529a48 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:15.188522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:15.190012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:15.217075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0e77b9-8bc8-4902-8741-0a49de3fa9d1 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:15.275080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398b6733-c814-4ef5-8e0c-39df4bebd0d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:15.275753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398b6733-c814-4ef5-8e0c-39df4bebd0d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:15.276068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398b6733-c814-4ef5-8e0c-39df4bebd0d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:15.276859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398b6733-c814-4ef5-8e0c-39df4bebd0d6 admin admin] PATH_INFO: `/v3/projects/a66d56505d08471894c9ca2b1bd179a3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:15.277995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398b6733-c814-4ef5-8e0c-39df4bebd0d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a66d56505d08471894c9ca2b1bd179a3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:15.280005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398b6733-c814-4ef5-8e0c-39df4bebd0d6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:15.404889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-398b6733-c814-4ef5-8e0c-39df4bebd0d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a66d56505d08471894c9ca2b1bd179a3', 'request_id': 'req-398b6733-c814-4ef5-8e0c-39df4bebd0d6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:15.421041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-398b6733-c814-4ef5-8e0c-39df4bebd0d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a66d56505d08471894c9ca2b1bd179a3', 'request_id': 'req-398b6733-c814-4ef5-8e0c-39df4bebd0d6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:15.427615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-398b6733-c814-4ef5-8e0c-39df4bebd0d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a66d56505d08471894c9ca2b1bd179a3', 'request_id': 'req-398b6733-c814-4ef5-8e0c-39df4bebd0d6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:15.523052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-398b6733-c814-4ef5-8e0c-39df4bebd0d6 admin admin] The token cache is being invalidate because project a66d56505d08471894c9ca2b1bd179a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:56:15.523356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-398b6733-c814-4ef5-8e0c-39df4bebd0d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:15.524060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-398b6733-c814-4ef5-8e0c-39df4bebd0d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:15.525600 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1180/2375] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:56:15 2022] DELETE /identity/v3/projects/a66d56505d08471894c9ca2b1bd179a3 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:15.531307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91920c56-4b05-43f0-abb7-c95feb36aa50 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:15.531830 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91920c56-4b05-43f0-abb7-c95feb36aa50 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:15.531942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91920c56-4b05-43f0-abb7-c95feb36aa50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:15.552270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-91920c56-4b05-43f0-abb7-c95feb36aa50 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:15.553291 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-91920c56-4b05-43f0-abb7-c95feb36aa50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:15.615662 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1196/2376] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:15.624065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6054d015-e701-4144-b9b1-f1cb1f07a913 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:15.624517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6054d015-e701-4144-b9b1-f1cb1f07a913 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:15.624668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6054d015-e701-4144-b9b1-f1cb1f07a913 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:15.632390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:15.645706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-6054d015-e701-4144-b9b1-f1cb1f07a913 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:15.647032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6054d015-e701-4144-b9b1-f1cb1f07a913 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:15.657573 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1181/2377] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:15.659858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91920c56-4b05-43f0-abb7-c95feb36aa50 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:15.669992 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91920c56-4b05-43f0-abb7-c95feb36aa50 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:15.674505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c1d98f-e74d-430b-b632-95d7bed2db4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:15.675090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c1d98f-e74d-430b-b632-95d7bed2db4e admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:15.675364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c1d98f-e74d-430b-b632-95d7bed2db4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:15.675630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c1d98f-e74d-430b-b632-95d7bed2db4e admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:15.676267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c1d98f-e74d-430b-b632-95d7bed2db4e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:56:15.676563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c1d98f-e74d-430b-b632-95d7bed2db4e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:15.678141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c1d98f-e74d-430b-b632-95d7bed2db4e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:15.692352 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1197/2378] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:56:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:56:15.696715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6054d015-e701-4144-b9b1-f1cb1f07a913 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:15.707141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6054d015-e701-4144-b9b1-f1cb1f07a913 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:15.712214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af4a132-fe98-4c10-a501-9d00b0ead5de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:15.712841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af4a132-fe98-4c10-a501-9d00b0ead5de admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:15.713153 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af4a132-fe98-4c10-a501-9d00b0ead5de admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:15.713557 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af4a132-fe98-4c10-a501-9d00b0ead5de admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:15.714211 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af4a132-fe98-4c10-a501-9d00b0ead5de admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:15.716152 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af4a132-fe98-4c10-a501-9d00b0ead5de admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:15.873849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1182/2379] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:56:15 2022] POST /identity/v3/projects => generated 383 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:15.878132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c1d98f-e74d-430b-b632-95d7bed2db4e admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:15.888853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c1d98f-e74d-430b-b632-95d7bed2db4e admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:15.945988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55407de-31ca-41d8-92c2-724ce288134e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:15.946539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55407de-31ca-41d8-92c2-724ce288134e admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:15.946817 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55407de-31ca-41d8-92c2-724ce288134e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:15.947085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55407de-31ca-41d8-92c2-724ce288134e admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:15.947854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55407de-31ca-41d8-92c2-724ce288134e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:15.949968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55407de-31ca-41d8-92c2-724ce288134e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:15.958657 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:16.069150 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1198/2380] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:56:15 2022] POST /identity/v3/users => generated 292 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:16.070346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af4a132-fe98-4c10-a501-9d00b0ead5de admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:16.082239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af4a132-fe98-4c10-a501-9d00b0ead5de admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:16.087234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb900eca-8578-4dc0-919e-fa97eb13bf2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:16.087854 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb900eca-8578-4dc0-919e-fa97eb13bf2a admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:16.088183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb900eca-8578-4dc0-919e-fa97eb13bf2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:16.088478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb900eca-8578-4dc0-919e-fa97eb13bf2a admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:16.089175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb900eca-8578-4dc0-919e-fa97eb13bf2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:16.090876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb900eca-8578-4dc0-919e-fa97eb13bf2a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:16.101935 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1183/2381] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:16.107526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55407de-31ca-41d8-92c2-724ce288134e admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:16.117561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55407de-31ca-41d8-92c2-724ce288134e admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:16.123189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ad03a4-5e3a-4ca6-adce-5f1149ea1fb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:16.123818 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ad03a4-5e3a-4ca6-adce-5f1149ea1fb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:16.124099 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ad03a4-5e3a-4ca6-adce-5f1149ea1fb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:16.124372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ad03a4-5e3a-4ca6-adce-5f1149ea1fb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:16.125016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ad03a4-5e3a-4ca6-adce-5f1149ea1fb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:16.126649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ad03a4-5e3a-4ca6-adce-5f1149ea1fb4 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:16.137906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1199/2382] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:16.142238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb900eca-8578-4dc0-919e-fa97eb13bf2a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:16.153253 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb900eca-8578-4dc0-919e-fa97eb13bf2a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:16.157997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5003f92-52ae-40a4-9ac5-60c83fc88e44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:16.158690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5003f92-52ae-40a4-9ac5-60c83fc88e44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:16.158976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5003f92-52ae-40a4-9ac5-60c83fc88e44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:16.159251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5003f92-52ae-40a4-9ac5-60c83fc88e44 admin admin] PATH_INFO: `/v3/projects/910d54fd0d0c4638a6998efc29e393a8/users/ccb3cd45120f45e9ac38d7489ca29a5c/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:16.175321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5003f92-52ae-40a4-9ac5-60c83fc88e44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=910d54fd0d0c4638a6998efc29e393a8, user_id=ccb3cd45120f45e9ac38d7489ca29a5c, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:16.178001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5003f92-52ae-40a4-9ac5-60c83fc88e44 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:16.260652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1184/2383] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:56:16 2022] PUT /identity/v3/projects/910d54fd0d0c4638a6998efc29e393a8/users/ccb3cd45120f45e9ac38d7489ca29a5c/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:16.271628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc40cbe6-b65e-4e50-bd3f-635a934d4823 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:16.271819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc40cbe6-b65e-4e50-bd3f-635a934d4823 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:16.272086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc40cbe6-b65e-4e50-bd3f-635a934d4823 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:16.302437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-bc40cbe6-b65e-4e50-bd3f-635a934d4823 None None] MFA Rules not processed for user `ccb3cd45120f45e9ac38d7489ca29a5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:16.303710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-bc40cbe6-b65e-4e50-bd3f-635a934d4823 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:16.378365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1200/2384] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:16 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:16.379711 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5003f92-52ae-40a4-9ac5-60c83fc88e44 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:16.447621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5003f92-52ae-40a4-9ac5-60c83fc88e44 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:16.452377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37a8df9-f155-4eaa-9e53-6cd53658b6c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:16.452986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37a8df9-f155-4eaa-9e53-6cd53658b6c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:16.453267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37a8df9-f155-4eaa-9e53-6cd53658b6c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:16.453540 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37a8df9-f155-4eaa-9e53-6cd53658b6c5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:16.454241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37a8df9-f155-4eaa-9e53-6cd53658b6c5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:16.456147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37a8df9-f155-4eaa-9e53-6cd53658b6c5 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:16.574455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1185/2385] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:56:16 2022] POST /identity/v3/projects => generated 383 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:16.578703 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc40cbe6-b65e-4e50-bd3f-635a934d4823 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:16.591062 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc40cbe6-b65e-4e50-bd3f-635a934d4823 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:16.653649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915b24e9-0979-4639-a3ec-65a641fdcb18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:16.654307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915b24e9-0979-4639-a3ec-65a641fdcb18 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:16.654589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915b24e9-0979-4639-a3ec-65a641fdcb18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:16.654861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915b24e9-0979-4639-a3ec-65a641fdcb18 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:16.655487 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915b24e9-0979-4639-a3ec-65a641fdcb18 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:16.657398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915b24e9-0979-4639-a3ec-65a641fdcb18 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:16.786496 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1201/2386] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:56:16 2022] POST /identity/v3/users => generated 298 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:16.790482 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37a8df9-f155-4eaa-9e53-6cd53658b6c5 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:16.801032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37a8df9-f155-4eaa-9e53-6cd53658b6c5 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:16.806081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04371f8d-5809-4d6f-9bc7-79bff16f7e52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:16.806709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04371f8d-5809-4d6f-9bc7-79bff16f7e52 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:16.807208 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04371f8d-5809-4d6f-9bc7-79bff16f7e52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:16.807352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04371f8d-5809-4d6f-9bc7-79bff16f7e52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:16.807971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04371f8d-5809-4d6f-9bc7-79bff16f7e52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:16.809644 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04371f8d-5809-4d6f-9bc7-79bff16f7e52 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:16.820950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1186/2387] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:16.827699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915b24e9-0979-4639-a3ec-65a641fdcb18 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:16.838128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915b24e9-0979-4639-a3ec-65a641fdcb18 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:16.843584 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b6f01b-d6bb-47fb-8fad-87d0a7f143dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:16.844239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b6f01b-d6bb-47fb-8fad-87d0a7f143dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:16.844543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b6f01b-d6bb-47fb-8fad-87d0a7f143dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:16.844816 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b6f01b-d6bb-47fb-8fad-87d0a7f143dc admin admin] PATH_INFO: `/v3/projects/90d62fe27ef34b09b159accd9c39b3d6/users/f55f09e14e3d4997bb958f0a16fe4148/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:16.860377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b6f01b-d6bb-47fb-8fad-87d0a7f143dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=90d62fe27ef34b09b159accd9c39b3d6, user_id=f55f09e14e3d4997bb958f0a16fe4148, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:16.863122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b6f01b-d6bb-47fb-8fad-87d0a7f143dc admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:16.961976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1202/2388] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:56:16 2022] PUT /identity/v3/projects/90d62fe27ef34b09b159accd9c39b3d6/users/f55f09e14e3d4997bb958f0a16fe4148/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:16.966569 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04371f8d-5809-4d6f-9bc7-79bff16f7e52 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:16.977323 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04371f8d-5809-4d6f-9bc7-79bff16f7e52 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:17.034690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722725c7-2370-4b75-9e08-1a3f486541a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:17.035287 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722725c7-2370-4b75-9e08-1a3f486541a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:17.035565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722725c7-2370-4b75-9e08-1a3f486541a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:17.035838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722725c7-2370-4b75-9e08-1a3f486541a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:17.036473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722725c7-2370-4b75-9e08-1a3f486541a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:17.038064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722725c7-2370-4b75-9e08-1a3f486541a0 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:17.047905 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1187/2389] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:16 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:17.052399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b6f01b-d6bb-47fb-8fad-87d0a7f143dc admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:17.063496 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b6f01b-d6bb-47fb-8fad-87d0a7f143dc admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:17.068148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0749520e-7456-4674-8ffc-fac340df47d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:17.068790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0749520e-7456-4674-8ffc-fac340df47d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:17.069068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0749520e-7456-4674-8ffc-fac340df47d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:17.069339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0749520e-7456-4674-8ffc-fac340df47d7 admin admin] PATH_INFO: `/v3/projects/90d62fe27ef34b09b159accd9c39b3d6/users/f55f09e14e3d4997bb958f0a16fe4148/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:17.071287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0749520e-7456-4674-8ffc-fac340df47d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=90d62fe27ef34b09b159accd9c39b3d6, user_id=f55f09e14e3d4997bb958f0a16fe4148, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:17.074585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0749520e-7456-4674-8ffc-fac340df47d7 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:17.085221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1203/2390] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:56:17 2022] PUT /identity/v3/projects/90d62fe27ef34b09b159accd9c39b3d6/users/f55f09e14e3d4997bb958f0a16fe4148/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:17.096945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1638602-db20-4fbb-bb7e-58f887bf89d0 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:17.097234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1638602-db20-4fbb-bb7e-58f887bf89d0 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:17.097510 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1638602-db20-4fbb-bb7e-58f887bf89d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:17.118066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:17.127522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-c1638602-db20-4fbb-bb7e-58f887bf89d0 None None] MFA Rules not processed for user `f55f09e14e3d4997bb958f0a16fe4148`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:17.131188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c1638602-db20-4fbb-bb7e-58f887bf89d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:17.194775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1188/2391] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:17 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:17.206406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0749520e-7456-4674-8ffc-fac340df47d7 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:17.208187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-0749520e-7456-4674-8ffc-fac340df47d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:17.235322 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:INSERT=7,keystone:SELECT=93 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:17.268569 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0749520e-7456-4674-8ffc-fac340df47d7 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:17.275827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e08d5a2-4353-4f84-be89-5e1798c04fbd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:17.276434 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e08d5a2-4353-4f84-be89-5e1798c04fbd service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:17.276707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e08d5a2-4353-4f84-be89-5e1798c04fbd service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:17.276976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e08d5a2-4353-4f84-be89-5e1798c04fbd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:17.286963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e08d5a2-4353-4f84-be89-5e1798c04fbd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:17.288389 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e08d5a2-4353-4f84-be89-5e1798c04fbd service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:17.298818 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1204/2392] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:17 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:56:17.564309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1638602-db20-4fbb-bb7e-58f887bf89d0 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:17.574692 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1638602-db20-4fbb-bb7e-58f887bf89d0 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:17.578430 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3f7523-4e6d-4a28-922e-61fe8b729bfa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:17.579032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3f7523-4e6d-4a28-922e-61fe8b729bfa service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:17.579309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3f7523-4e6d-4a28-922e-61fe8b729bfa service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:17.579580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3f7523-4e6d-4a28-922e-61fe8b729bfa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:17.640171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3f7523-4e6d-4a28-922e-61fe8b729bfa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:17.641709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3f7523-4e6d-4a28-922e-61fe8b729bfa service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:17.651766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1189/2393] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:17 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:56:20.939781 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e08d5a2-4353-4f84-be89-5e1798c04fbd service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:21.008386 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e08d5a2-4353-4f84-be89-5e1798c04fbd service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:21.013878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:21.014542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:21.014820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:21.015091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] PATH_INFO: `/v3/users/ccb3cd45120f45e9ac38d7489ca29a5c` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:21.016214 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ccb3cd45120f45e9ac38d7489ca29a5c)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:21.018192 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:21.240917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ccb3cd45120f45e9ac38d7489ca29a5c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:21.241545 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ccb3cd45120f45e9ac38d7489ca29a5c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:21.249379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ccb3cd45120f45e9ac38d7489ca29a5c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:21.250016 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ccb3cd45120f45e9ac38d7489ca29a5c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:21.332900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ccb3cd45120f45e9ac38d7489ca29a5c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:21.351502 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1205/2394] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:56:20 2022] DELETE /identity/v3/users/ccb3cd45120f45e9ac38d7489ca29a5c => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:21.357391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246288c5-3e4f-4ca7-addb-f6559693db2e None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:21.358339 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246288c5-3e4f-4ca7-addb-f6559693db2e None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:21.358911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246288c5-3e4f-4ca7-addb-f6559693db2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:21.384845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-246288c5-3e4f-4ca7-addb-f6559693db2e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:21.386808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-246288c5-3e4f-4ca7-addb-f6559693db2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:21.482193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1190/2395] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:21.490433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:21.491952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:21.547412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:21.560152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1cbc8a-1fdd-44ec-bece-bc4ae4c57981 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:21.571820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dc5fae-d5dc-4449-8940-213d9f6a15e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:21.571820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc5fae-d5dc-4449-8940-213d9f6a15e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:21.571820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc5fae-d5dc-4449-8940-213d9f6a15e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:21.571820 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc5fae-d5dc-4449-8940-213d9f6a15e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:21.575622 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dc5fae-d5dc-4449-8940-213d9f6a15e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:21.577068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dc5fae-d5dc-4449-8940-213d9f6a15e8 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:21.586675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1206/2396] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:56:22.012860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246288c5-3e4f-4ca7-addb-f6559693db2e None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:22.013016 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-246288c5-3e4f-4ca7-addb-f6559693db2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:22.027738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246288c5-3e4f-4ca7-addb-f6559693db2e None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:22.033542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:22.034629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:22.035123 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:22.035604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] PATH_INFO: `/v3/projects/910d54fd0d0c4638a6998efc29e393a8` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:22.037594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=910d54fd0d0c4638a6998efc29e393a8)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:22.042029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:22.159725 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:22.177066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '910d54fd0d0c4638a6998efc29e393a8', 'request_id': 'req-375e58e1-6f0b-4941-9413-accfb70953b6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:22.181788 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '910d54fd0d0c4638a6998efc29e393a8', 'request_id': 'req-375e58e1-6f0b-4941-9413-accfb70953b6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:22.188063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '910d54fd0d0c4638a6998efc29e393a8', 'request_id': 'req-375e58e1-6f0b-4941-9413-accfb70953b6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:22.276975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] The token cache is being invalidate because project 910d54fd0d0c4638a6998efc29e393a8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:56:22.277264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:22.277896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:22.279424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1191/2397] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:56:22 2022] DELETE /identity/v3/projects/910d54fd0d0c4638a6998efc29e393a8 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:22.283383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dc5fae-d5dc-4449-8940-213d9f6a15e8 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:22.293942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dc5fae-d5dc-4449-8940-213d9f6a15e8 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:22.353294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6837a748-0f2e-44dc-a4e8-732d413793b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:22.354845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837a748-0f2e-44dc-a4e8-732d413793b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:22.355154 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837a748-0f2e-44dc-a4e8-732d413793b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:22.355433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837a748-0f2e-44dc-a4e8-732d413793b1 admin admin] PATH_INFO: `/v3/users/f55f09e14e3d4997bb958f0a16fe4148` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:22.356682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6837a748-0f2e-44dc-a4e8-732d413793b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f55f09e14e3d4997bb958f0a16fe4148)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:22.358672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6837a748-0f2e-44dc-a4e8-732d413793b1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:22.549013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6837a748-0f2e-44dc-a4e8-732d413793b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f55f09e14e3d4997bb958f0a16fe4148'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:22.549624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6837a748-0f2e-44dc-a4e8-732d413793b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f55f09e14e3d4997bb958f0a16fe4148'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:22.557330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6837a748-0f2e-44dc-a4e8-732d413793b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f55f09e14e3d4997bb958f0a16fe4148'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:22.557639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6837a748-0f2e-44dc-a4e8-732d413793b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f55f09e14e3d4997bb958f0a16fe4148'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:22.625950 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-6837a748-0f2e-44dc-a4e8-732d413793b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f55f09e14e3d4997bb958f0a16fe4148'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:22.645563 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1207/2398] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:56:22 2022] DELETE /identity/v3/users/f55f09e14e3d4997bb958f0a16fe4148 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:23.101944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:23.104756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:23.117707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375e58e1-6f0b-4941-9413-accfb70953b6 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:23.181482 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5a1c43-ea74-4791-868a-167c7f53cad3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:23.182350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5a1c43-ea74-4791-868a-167c7f53cad3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:23.182966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5a1c43-ea74-4791-868a-167c7f53cad3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:23.182966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5a1c43-ea74-4791-868a-167c7f53cad3 admin admin] PATH_INFO: `/v3/projects/90d62fe27ef34b09b159accd9c39b3d6` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:23.184067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5a1c43-ea74-4791-868a-167c7f53cad3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=90d62fe27ef34b09b159accd9c39b3d6)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:23.186002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5a1c43-ea74-4791-868a-167c7f53cad3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:23.335069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6f5a1c43-ea74-4791-868a-167c7f53cad3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '90d62fe27ef34b09b159accd9c39b3d6', 'request_id': 'req-6f5a1c43-ea74-4791-868a-167c7f53cad3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:23.341437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6f5a1c43-ea74-4791-868a-167c7f53cad3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '90d62fe27ef34b09b159accd9c39b3d6', 'request_id': 'req-6f5a1c43-ea74-4791-868a-167c7f53cad3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:23.348022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6f5a1c43-ea74-4791-868a-167c7f53cad3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '90d62fe27ef34b09b159accd9c39b3d6', 'request_id': 'req-6f5a1c43-ea74-4791-868a-167c7f53cad3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:23.437608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6f5a1c43-ea74-4791-868a-167c7f53cad3 admin admin] The token cache is being invalidate because project 90d62fe27ef34b09b159accd9c39b3d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:56:23.441675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6f5a1c43-ea74-4791-868a-167c7f53cad3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:23.441675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6f5a1c43-ea74-4791-868a-167c7f53cad3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:23.441675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1192/2399] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:56:23 2022] DELETE /identity/v3/projects/90d62fe27ef34b09b159accd9c39b3d6 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:23.445932 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29addb7-be9b-48f8-94d0-2e3808f4dc3f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:23.446409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29addb7-be9b-48f8-94d0-2e3808f4dc3f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:23.446535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29addb7-be9b-48f8-94d0-2e3808f4dc3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:23.470289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-d29addb7-be9b-48f8-94d0-2e3808f4dc3f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:23.476187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d29addb7-be9b-48f8-94d0-2e3808f4dc3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:23.541729 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1208/2400] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:23.549591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9bc0ac-0de2-4182-ae2d-d23f294b7ec6 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:23.549881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9bc0ac-0de2-4182-ae2d-d23f294b7ec6 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:23.550181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9bc0ac-0de2-4182-ae2d-d23f294b7ec6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:23.570949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-aa9bc0ac-0de2-4182-ae2d-d23f294b7ec6 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:23.572184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-aa9bc0ac-0de2-4182-ae2d-d23f294b7ec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:23.580989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1193/2401] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:23.588856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29addb7-be9b-48f8-94d0-2e3808f4dc3f None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:23.598649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29addb7-be9b-48f8-94d0-2e3808f4dc3f None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:23.603407 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55add9c2-953f-4144-a6b8-a764212a9113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:23.604886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55add9c2-953f-4144-a6b8-a764212a9113 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:23.605202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55add9c2-953f-4144-a6b8-a764212a9113 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:23.605469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55add9c2-953f-4144-a6b8-a764212a9113 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:23.606113 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55add9c2-953f-4144-a6b8-a764212a9113 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:56:23.606443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55add9c2-953f-4144-a6b8-a764212a9113 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:23.608038 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55add9c2-953f-4144-a6b8-a764212a9113 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:23.609818 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:23.621836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1209/2402] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:56:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:56:23.626561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9bc0ac-0de2-4182-ae2d-d23f294b7ec6 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:23.636653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9bc0ac-0de2-4182-ae2d-d23f294b7ec6 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:23.641588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2f5cce-5ebf-4ce1-97ac-c1142ec51e13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:23.642224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2f5cce-5ebf-4ce1-97ac-c1142ec51e13 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:23.642506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2f5cce-5ebf-4ce1-97ac-c1142ec51e13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:23.642783 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2f5cce-5ebf-4ce1-97ac-c1142ec51e13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:23.643409 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2f5cce-5ebf-4ce1-97ac-c1142ec51e13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:23.645542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2f5cce-5ebf-4ce1-97ac-c1142ec51e13 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:23.795780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1194/2403] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:56:23 2022] POST /identity/v3/projects => generated 383 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:23.796414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55add9c2-953f-4144-a6b8-a764212a9113 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:23.809848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55add9c2-953f-4144-a6b8-a764212a9113 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:23.875691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95e661d-59ae-40c4-9a94-820bf91653e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:23.876311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95e661d-59ae-40c4-9a94-820bf91653e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:23.879309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95e661d-59ae-40c4-9a94-820bf91653e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:23.879309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95e661d-59ae-40c4-9a94-820bf91653e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:23.879309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95e661d-59ae-40c4-9a94-820bf91653e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:23.879607 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95e661d-59ae-40c4-9a94-820bf91653e1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:23.976275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1210/2404] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:56:23 2022] POST /identity/v3/users => generated 292 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:23.977498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2f5cce-5ebf-4ce1-97ac-c1142ec51e13 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:23.989017 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2f5cce-5ebf-4ce1-97ac-c1142ec51e13 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:23.993835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce22811c-84f1-4ca5-b11d-dc9c76a83ded admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:23.994479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce22811c-84f1-4ca5-b11d-dc9c76a83ded admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:23.994757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce22811c-84f1-4ca5-b11d-dc9c76a83ded admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:23.995022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce22811c-84f1-4ca5-b11d-dc9c76a83ded admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:23.995664 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce22811c-84f1-4ca5-b11d-dc9c76a83ded admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:23.997244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce22811c-84f1-4ca5-b11d-dc9c76a83ded admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:24.007468 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1195/2405] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:24.011689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95e661d-59ae-40c4-9a94-820bf91653e1 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:24.021565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95e661d-59ae-40c4-9a94-820bf91653e1 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:24.026945 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b33734-9193-4dcc-a095-796f8cb2d040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:24.031226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b33734-9193-4dcc-a095-796f8cb2d040 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:24.031226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b33734-9193-4dcc-a095-796f8cb2d040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:24.031226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b33734-9193-4dcc-a095-796f8cb2d040 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:24.031226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b33734-9193-4dcc-a095-796f8cb2d040 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:24.031226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b33734-9193-4dcc-a095-796f8cb2d040 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:24.040995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1211/2406] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:24.045954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce22811c-84f1-4ca5-b11d-dc9c76a83ded admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:24.056039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce22811c-84f1-4ca5-b11d-dc9c76a83ded admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:24.060812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a88b9e-3351-46e6-aea3-693e9396f3c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:24.068766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a88b9e-3351-46e6-aea3-693e9396f3c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:24.068766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a88b9e-3351-46e6-aea3-693e9396f3c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:24.068766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a88b9e-3351-46e6-aea3-693e9396f3c5 admin admin] PATH_INFO: `/v3/projects/6964f1d0e825442d8ca423be7bc06139/users/8e0e11414fa34780b5a2725a90fe4526/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:24.068766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:24.077472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a88b9e-3351-46e6-aea3-693e9396f3c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6964f1d0e825442d8ca423be7bc06139, user_id=8e0e11414fa34780b5a2725a90fe4526, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:24.080158 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a88b9e-3351-46e6-aea3-693e9396f3c5 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:24.151693 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1196/2407] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:56:24 2022] PUT /identity/v3/projects/6964f1d0e825442d8ca423be7bc06139/users/8e0e11414fa34780b5a2725a90fe4526/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:24.163296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e2808-f629-4cc0-b00c-82d669a88b3d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:24.166235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e2808-f629-4cc0-b00c-82d669a88b3d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:24.166235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e2808-f629-4cc0-b00c-82d669a88b3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:24.192789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c34e2808-f629-4cc0-b00c-82d669a88b3d None None] MFA Rules not processed for user `8e0e11414fa34780b5a2725a90fe4526`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:24.194239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c34e2808-f629-4cc0-b00c-82d669a88b3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:24.264588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1212/2408] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:56:24.269395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a88b9e-3351-46e6-aea3-693e9396f3c5 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:24.332354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a88b9e-3351-46e6-aea3-693e9396f3c5 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:24.337006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a96ef21-90ba-4193-a378-2e6b519d5f90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:24.337599 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a96ef21-90ba-4193-a378-2e6b519d5f90 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:24.337872 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a96ef21-90ba-4193-a378-2e6b519d5f90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:24.338135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a96ef21-90ba-4193-a378-2e6b519d5f90 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:24.339495 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a96ef21-90ba-4193-a378-2e6b519d5f90 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:24.341749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a96ef21-90ba-4193-a378-2e6b519d5f90 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:24.404812 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a02251-b9a0-48f2-b0ab-93eba906cf1d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:24.405110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a02251-b9a0-48f2-b0ab-93eba906cf1d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:24.405380 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a02251-b9a0-48f2-b0ab-93eba906cf1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:24.437047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-35a02251-b9a0-48f2-b0ab-93eba906cf1d None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:24.441082 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-35a02251-b9a0-48f2-b0ab-93eba906cf1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:24.476809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1197/2409] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:56:24 2022] POST /identity/v3/projects => generated 383 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:24.477860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a96ef21-90ba-4193-a378-2e6b519d5f90 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:24.488202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:24.497636 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1213/2410] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:56:24 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:56:24.545634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a96ef21-90ba-4193-a378-2e6b519d5f90 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:24.550869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a0cdd3-8cf7-43f9-b13d-c68d0422cc13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:24.551531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a0cdd3-8cf7-43f9-b13d-c68d0422cc13 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:24.555738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a0cdd3-8cf7-43f9-b13d-c68d0422cc13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:24.555738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a0cdd3-8cf7-43f9-b13d-c68d0422cc13 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:24.555738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a0cdd3-8cf7-43f9-b13d-c68d0422cc13 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:24.555738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a0cdd3-8cf7-43f9-b13d-c68d0422cc13 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:24.683626 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a02251-b9a0-48f2-b0ab-93eba906cf1d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:24.687918 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1198/2411] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:56:24 2022] POST /identity/v3/users => generated 292 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:24.688061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-35a02251-b9a0-48f2-b0ab-93eba906cf1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:24.692300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a0cdd3-8cf7-43f9-b13d-c68d0422cc13 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:24.705219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a0cdd3-8cf7-43f9-b13d-c68d0422cc13 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:24.712190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7454ad2-3a28-4758-9e6d-8a353d79ad57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:24.712797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7454ad2-3a28-4758-9e6d-8a353d79ad57 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:24.713082 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7454ad2-3a28-4758-9e6d-8a353d79ad57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:24.713355 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7454ad2-3a28-4758-9e6d-8a353d79ad57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:24.714064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7454ad2-3a28-4758-9e6d-8a353d79ad57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:24.715759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7454ad2-3a28-4758-9e6d-8a353d79ad57 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:24.727541 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1199/2412] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:24 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:56:24.732078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7454ad2-3a28-4758-9e6d-8a353d79ad57 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:24.741940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7454ad2-3a28-4758-9e6d-8a353d79ad57 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:24.747305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ec64c2-f8d5-48c5-b439-33a4e9ad6174 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:24.747886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ec64c2-f8d5-48c5-b439-33a4e9ad6174 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:24.748188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ec64c2-f8d5-48c5-b439-33a4e9ad6174 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:24.748458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ec64c2-f8d5-48c5-b439-33a4e9ad6174 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:24.749085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ec64c2-f8d5-48c5-b439-33a4e9ad6174 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:24.750906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ec64c2-f8d5-48c5-b439-33a4e9ad6174 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:24.761895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1200/2413] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:56:24.766029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ec64c2-f8d5-48c5-b439-33a4e9ad6174 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:24.770901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a02251-b9a0-48f2-b0ab-93eba906cf1d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:24.775423 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ec64c2-f8d5-48c5-b439-33a4e9ad6174 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:24.780267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5df5e2-0cba-414c-8a23-ae3d6393ca2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:24.780910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5df5e2-0cba-414c-8a23-ae3d6393ca2e admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:24.781181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5df5e2-0cba-414c-8a23-ae3d6393ca2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:24.781445 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5df5e2-0cba-414c-8a23-ae3d6393ca2e admin admin] PATH_INFO: `/v3/projects/d92ec38fb36c4b318314153ba9b788dc/users/dc33ee5f7e3e4966a31bc1ad042c0db1/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:24.798607 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5df5e2-0cba-414c-8a23-ae3d6393ca2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d92ec38fb36c4b318314153ba9b788dc, user_id=dc33ee5f7e3e4966a31bc1ad042c0db1, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:24.801300 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5df5e2-0cba-414c-8a23-ae3d6393ca2e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:24.839846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ace3b13-bcdd-4307-90e4-0819e9627d9a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:24.840487 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ace3b13-bcdd-4307-90e4-0819e9627d9a service swift] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:24.840773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ace3b13-bcdd-4307-90e4-0819e9627d9a service swift] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:24.841044 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ace3b13-bcdd-4307-90e4-0819e9627d9a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:24.888014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1201/2414] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:56:24 2022] PUT /identity/v3/projects/d92ec38fb36c4b318314153ba9b788dc/users/dc33ee5f7e3e4966a31bc1ad042c0db1/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:24.920318 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ace3b13-bcdd-4307-90e4-0819e9627d9a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:24.922506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ace3b13-bcdd-4307-90e4-0819e9627d9a service swift] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:24.936260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1214/2415] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:56:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 254 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:56:24.961577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10763673-1513-459e-9c65-04060dfa3e28 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:24.961896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10763673-1513-459e-9c65-04060dfa3e28 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:24.962207 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10763673-1513-459e-9c65-04060dfa3e28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:24.990980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:25.008447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-10763673-1513-459e-9c65-04060dfa3e28 None None] MFA Rules not processed for user `dc33ee5f7e3e4966a31bc1ad042c0db1`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:25.016209 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-10763673-1513-459e-9c65-04060dfa3e28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:25.113382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ace3b13-bcdd-4307-90e4-0819e9627d9a service swift] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:25.120576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1202/2416] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:25.131465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ace3b13-bcdd-4307-90e4-0819e9627d9a service swift] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:25.184000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:25.206885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caea9711-52a0-43f9-a021-72475304ee8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:25.207606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caea9711-52a0-43f9-a021-72475304ee8f admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:25.207941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caea9711-52a0-43f9-a021-72475304ee8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:25.208300 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caea9711-52a0-43f9-a021-72475304ee8f admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:25.208973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caea9711-52a0-43f9-a021-72475304ee8f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:25.211013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caea9711-52a0-43f9-a021-72475304ee8f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:25.310645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1215/2417] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:56:25 2022] POST /identity/v3/projects => generated 383 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:25.313384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10763673-1513-459e-9c65-04060dfa3e28 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:25.326837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10763673-1513-459e-9c65-04060dfa3e28 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:25.409820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d5d748-737d-4153-98f0-b19b21804f59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:25.411328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d5d748-737d-4153-98f0-b19b21804f59 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:25.411836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d5d748-737d-4153-98f0-b19b21804f59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:25.412532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d5d748-737d-4153-98f0-b19b21804f59 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:25.413356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d5d748-737d-4153-98f0-b19b21804f59 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:25.415720 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d5d748-737d-4153-98f0-b19b21804f59 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:25.541217 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1203/2418] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:56:25 2022] POST /identity/v3/users => generated 298 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:56:25.545803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caea9711-52a0-43f9-a021-72475304ee8f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:25.560614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caea9711-52a0-43f9-a021-72475304ee8f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:25.567088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54d37f-1263-47fc-a9dd-556607cc88e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:25.576724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54d37f-1263-47fc-a9dd-556607cc88e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:25.576724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54d37f-1263-47fc-a9dd-556607cc88e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:25.576724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54d37f-1263-47fc-a9dd-556607cc88e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:25.576724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c54d37f-1263-47fc-a9dd-556607cc88e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:25.576724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c54d37f-1263-47fc-a9dd-556607cc88e0 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:25.590193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1216/2419] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:25.592858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d5d748-737d-4153-98f0-b19b21804f59 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:25.605603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d5d748-737d-4153-98f0-b19b21804f59 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:25.611066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ba1eb3-b418-401e-a197-98b456b9ca88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:25.611815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ba1eb3-b418-401e-a197-98b456b9ca88 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:25.612137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ba1eb3-b418-401e-a197-98b456b9ca88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:25.612416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ba1eb3-b418-401e-a197-98b456b9ca88 admin admin] PATH_INFO: `/v3/projects/d09bbccb062e40adbc66f8751204cf3b/users/a399661339c8499ea8e5982a34bb4a1a/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:25.630556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ba1eb3-b418-401e-a197-98b456b9ca88 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d09bbccb062e40adbc66f8751204cf3b, user_id=a399661339c8499ea8e5982a34bb4a1a, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:25.633225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ba1eb3-b418-401e-a197-98b456b9ca88 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:25.758474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1204/2420] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:56:25 2022] PUT /identity/v3/projects/d09bbccb062e40adbc66f8751204cf3b/users/a399661339c8499ea8e5982a34bb4a1a/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:56:25.759580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54d37f-1263-47fc-a9dd-556607cc88e0 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:25.772195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54d37f-1263-47fc-a9dd-556607cc88e0 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:25.845158 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de6027a-9d5b-447a-b5ea-567b99265994 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:25.845849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de6027a-9d5b-447a-b5ea-567b99265994 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:25.846091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de6027a-9d5b-447a-b5ea-567b99265994 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:25.846348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de6027a-9d5b-447a-b5ea-567b99265994 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:25.851885 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de6027a-9d5b-447a-b5ea-567b99265994 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:25.854249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de6027a-9d5b-447a-b5ea-567b99265994 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:25.868581 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1217/2421] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:56:25.872676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ba1eb3-b418-401e-a197-98b456b9ca88 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:25.890793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ba1eb3-b418-401e-a197-98b456b9ca88 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:25.896746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c123c2-9698-47e2-98c0-101c5243ea3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:25.897416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c123c2-9698-47e2-98c0-101c5243ea3a admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:25.897691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c123c2-9698-47e2-98c0-101c5243ea3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:25.897960 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c123c2-9698-47e2-98c0-101c5243ea3a admin admin] PATH_INFO: `/v3/projects/d09bbccb062e40adbc66f8751204cf3b/users/a399661339c8499ea8e5982a34bb4a1a/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:25.899961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c123c2-9698-47e2-98c0-101c5243ea3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d09bbccb062e40adbc66f8751204cf3b, user_id=a399661339c8499ea8e5982a34bb4a1a, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:25.903485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c123c2-9698-47e2-98c0-101c5243ea3a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:25.934759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1205/2422] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:56:25 2022] PUT /identity/v3/projects/d09bbccb062e40adbc66f8751204cf3b/users/a399661339c8499ea8e5982a34bb4a1a/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:25.945696 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e639505-8e42-4ff6-9be6-486f9bfcb462 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:25.945991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e639505-8e42-4ff6-9be6-486f9bfcb462 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:25.946278 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e639505-8e42-4ff6-9be6-486f9bfcb462 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:25.978593 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5e639505-8e42-4ff6-9be6-486f9bfcb462 None None] MFA Rules not processed for user `a399661339c8499ea8e5982a34bb4a1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:25.980168 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5e639505-8e42-4ff6-9be6-486f9bfcb462 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:26.060706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1218/2423] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:25 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:56:26.068564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c123c2-9698-47e2-98c0-101c5243ea3a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:26.070389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-23c123c2-9698-47e2-98c0-101c5243ea3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:26.167471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c123c2-9698-47e2-98c0-101c5243ea3a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:26.189333 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd96ba87-94fa-43f0-b185-59b325c7e680 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:26.191167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd96ba87-94fa-43f0-b185-59b325c7e680 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:26.192327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd96ba87-94fa-43f0-b185-59b325c7e680 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:26.192880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd96ba87-94fa-43f0-b185-59b325c7e680 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:26.201679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:26.270244 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd96ba87-94fa-43f0-b185-59b325c7e680 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:26.271755 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd96ba87-94fa-43f0-b185-59b325c7e680 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:26.290401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1206/2424] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:26 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:56:29.934602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e639505-8e42-4ff6-9be6-486f9bfcb462 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:29.951330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e639505-8e42-4ff6-9be6-486f9bfcb462 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:29.958948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc494f4c-1c22-4f23-b7f3-b621fb5f4bcf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:29.959580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc494f4c-1c22-4f23-b7f3-b621fb5f4bcf service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:29.959853 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc494f4c-1c22-4f23-b7f3-b621fb5f4bcf service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:29.960123 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc494f4c-1c22-4f23-b7f3-b621fb5f4bcf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:30.078947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:30.113153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc494f4c-1c22-4f23-b7f3-b621fb5f4bcf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:30.115402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc494f4c-1c22-4f23-b7f3-b621fb5f4bcf service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:30.144505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1219/2425] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:29 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:56:31.943316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd96ba87-94fa-43f0-b185-59b325c7e680 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:31.944762 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-bd96ba87-94fa-43f0-b185-59b325c7e680 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:31.959805 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd96ba87-94fa-43f0-b185-59b325c7e680 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:31.963600 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248bc5c8-567a-4e7f-aa92-1aae419197f1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:31.964224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248bc5c8-567a-4e7f-aa92-1aae419197f1 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:31.964490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248bc5c8-567a-4e7f-aa92-1aae419197f1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:31.964760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248bc5c8-567a-4e7f-aa92-1aae419197f1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:31.974959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248bc5c8-567a-4e7f-aa92-1aae419197f1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:31.976669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248bc5c8-567a-4e7f-aa92-1aae419197f1 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:31.987646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1207/2426] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:31 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:56:37.401685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc494f4c-1c22-4f23-b7f3-b621fb5f4bcf service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:37.469869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc494f4c-1c22-4f23-b7f3-b621fb5f4bcf service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:37.474353 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:37.474976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:37.475249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:37.475515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] PATH_INFO: `/v3/users/8e0e11414fa34780b5a2725a90fe4526` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:37.476628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e0e11414fa34780b5a2725a90fe4526)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:37.478596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:37.764220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e0e11414fa34780b5a2725a90fe4526'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:37.764888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e0e11414fa34780b5a2725a90fe4526'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:37.772033 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e0e11414fa34780b5a2725a90fe4526'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:37.772724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e0e11414fa34780b5a2725a90fe4526'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:37.888576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e0e11414fa34780b5a2725a90fe4526'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:37.907262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1220/2427] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:56:37 2022] DELETE /identity/v3/users/8e0e11414fa34780b5a2725a90fe4526 => generated 0 bytes in 506 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:37.911838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2037c4bd-ecd9-4806-9cd4-64fa2af3dbbc None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:37.914256 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2037c4bd-ecd9-4806-9cd4-64fa2af3dbbc None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:37.914256 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2037c4bd-ecd9-4806-9cd4-64fa2af3dbbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:37.931514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-2037c4bd-ecd9-4806-9cd4-64fa2af3dbbc None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:37.932782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-2037c4bd-ecd9-4806-9cd4-64fa2af3dbbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:37.998112 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1208/2428] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:56:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:56:38.007909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:38.009345 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:38.070284 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:38.091097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c431b42e-c073-4d6a-b765-db704385405a admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:38.094761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8edf107-7e57-4029-8ffc-f95bd4a6d22e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:38.095364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8edf107-7e57-4029-8ffc-f95bd4a6d22e service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:38.095644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8edf107-7e57-4029-8ffc-f95bd4a6d22e service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:38.095915 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8edf107-7e57-4029-8ffc-f95bd4a6d22e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:38.105898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8edf107-7e57-4029-8ffc-f95bd4a6d22e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:38.107397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8edf107-7e57-4029-8ffc-f95bd4a6d22e service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:38.117800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1221/2429] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:56:38.665396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2037c4bd-ecd9-4806-9cd4-64fa2af3dbbc None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:38.667112 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-2037c4bd-ecd9-4806-9cd4-64fa2af3dbbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:38.686472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2037c4bd-ecd9-4806-9cd4-64fa2af3dbbc None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:38.691598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:38.692251 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:38.692526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:38.692794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] PATH_INFO: `/v3/projects/6964f1d0e825442d8ca423be7bc06139` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:38.694059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6964f1d0e825442d8ca423be7bc06139)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:38.696010 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:38.814682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6964f1d0e825442d8ca423be7bc06139', 'request_id': 'req-84a23922-af76-4040-8f49-ad6a8afd7561'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:38.820161 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6964f1d0e825442d8ca423be7bc06139', 'request_id': 'req-84a23922-af76-4040-8f49-ad6a8afd7561'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:38.826919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6964f1d0e825442d8ca423be7bc06139', 'request_id': 'req-84a23922-af76-4040-8f49-ad6a8afd7561'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:38.923232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] The token cache is being invalidate because project 6964f1d0e825442d8ca423be7bc06139 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:56:38.923940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:38.924366 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:38.926531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1209/2430] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:56:38 2022] DELETE /identity/v3/projects/6964f1d0e825442d8ca423be7bc06139 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:38.930658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8edf107-7e57-4029-8ffc-f95bd4a6d22e service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:38.941188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8edf107-7e57-4029-8ffc-f95bd4a6d22e service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:38.999481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:39.000109 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:39.000418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:39.000692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] PATH_INFO: `/v3/users/dc33ee5f7e3e4966a31bc1ad042c0db1` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:39.001939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc33ee5f7e3e4966a31bc1ad042c0db1)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:39.003972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:39.176028 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc33ee5f7e3e4966a31bc1ad042c0db1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:39.176685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc33ee5f7e3e4966a31bc1ad042c0db1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:39.183734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc33ee5f7e3e4966a31bc1ad042c0db1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:39.185973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc33ee5f7e3e4966a31bc1ad042c0db1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:39.253874 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc33ee5f7e3e4966a31bc1ad042c0db1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:39.272541 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1222/2431] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:56:38 2022] DELETE /identity/v3/users/dc33ee5f7e3e4966a31bc1ad042c0db1 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:39.847351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:39.849227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:39.863450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a23922-af76-4040-8f49-ad6a8afd7561 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:39.878326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:39.922041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:39.922696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:39.922979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:39.923254 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] PATH_INFO: `/v3/projects/d92ec38fb36c4b318314153ba9b788dc` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:39.924449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d92ec38fb36c4b318314153ba9b788dc)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:39.926479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:40.065183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd92ec38fb36c4b318314153ba9b788dc', 'request_id': 'req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:40.070623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd92ec38fb36c4b318314153ba9b788dc', 'request_id': 'req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:40.077881 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd92ec38fb36c4b318314153ba9b788dc', 'request_id': 'req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:40.202399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] The token cache is being invalidate because project d92ec38fb36c4b318314153ba9b788dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:56:40.202690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:40.203349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:40.204923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1210/2432] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:56:39 2022] DELETE /identity/v3/projects/d92ec38fb36c4b318314153ba9b788dc => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:40.209115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:40.210564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:40.223798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e98812d-87b2-4461-a9ef-7354e4722420 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:40.276513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:40.282566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:40.283220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:40.283530 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:40.283802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] PATH_INFO: `/v3/users/a399661339c8499ea8e5982a34bb4a1a` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:40.284958 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] RBAC: Authorizing `identity:delete_user(user_id=a399661339c8499ea8e5982a34bb4a1a)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:40.286947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:40.523469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a399661339c8499ea8e5982a34bb4a1a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:40.524135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a399661339c8499ea8e5982a34bb4a1a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:40.531210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a399661339c8499ea8e5982a34bb4a1a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:40.531860 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a399661339c8499ea8e5982a34bb4a1a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:40.623660 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a399661339c8499ea8e5982a34bb4a1a'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:40.642223 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1223/2433] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:56:40 2022] DELETE /identity/v3/users/a399661339c8499ea8e5982a34bb4a1a => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:41.272950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:41.274512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:41.287900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8e3fbd-b1d4-4609-be26-2c655e5c3d65 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:41.361678 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:41.363490 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:41.369835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:41.369835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:41.369835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:41.369835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] PATH_INFO: `/v3/projects/d09bbccb062e40adbc66f8751204cf3b` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:41.370245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d09bbccb062e40adbc66f8751204cf3b)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:41.372075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:41.468163 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896d8666-0595-463f-a1f4-d61a0b166a7a admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:41.473655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985a1a2b-fcee-4b31-8a20-9e73fa5f5d50 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:41.474522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985a1a2b-fcee-4b31-8a20-9e73fa5f5d50 service placement] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:41.474822 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985a1a2b-fcee-4b31-8a20-9e73fa5f5d50 service placement] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:41.475092 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985a1a2b-fcee-4b31-8a20-9e73fa5f5d50 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:41.476856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-985a1a2b-fcee-4b31-8a20-9e73fa5f5d50 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:41.490617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985a1a2b-fcee-4b31-8a20-9e73fa5f5d50 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:41.492306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985a1a2b-fcee-4b31-8a20-9e73fa5f5d50 service placement] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:41.503454 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1224/2434] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:56:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:56:41.507127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd09bbccb062e40adbc66f8751204cf3b', 'request_id': 'req-e368de2c-3730-40c2-a0c7-b6c3bc824223'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:41.508577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:41.515495 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd09bbccb062e40adbc66f8751204cf3b', 'request_id': 'req-e368de2c-3730-40c2-a0c7-b6c3bc824223'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:41.522615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd09bbccb062e40adbc66f8751204cf3b', 'request_id': 'req-e368de2c-3730-40c2-a0c7-b6c3bc824223'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:41.598492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] The token cache is being invalidate because project d09bbccb062e40adbc66f8751204cf3b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:56:41.598779 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:41.599484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:56:41.601501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1211/2435] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:56:41 2022] DELETE /identity/v3/projects/d09bbccb062e40adbc66f8751204cf3b => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:56:47.660898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4adcad8-302b-48bb-a291-0c4375c2db11 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:47.661348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4adcad8-302b-48bb-a291-0c4375c2db11 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:47.663210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4adcad8-302b-48bb-a291-0c4375c2db11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:47.680751 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-b4adcad8-302b-48bb-a291-0c4375c2db11 None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:56:47.681948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b4adcad8-302b-48bb-a291-0c4375c2db11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:47.723969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:47.737535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1225/2436] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 03:56:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:56:47.750564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:56:47.751957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:56:47.820906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e368de2c-3730-40c2-a0c7-b6c3bc824223 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:56:47.825888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0069ba71-a8d0-4937-9d0c-2576e6a2c115 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:56:47.826515 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0069ba71-a8d0-4937-9d0c-2576e6a2c115 service swift] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:56:47.827429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0069ba71-a8d0-4937-9d0c-2576e6a2c115 service swift] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:56:47.827907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0069ba71-a8d0-4937-9d0c-2576e6a2c115 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:56:47.835929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0069ba71-a8d0-4937-9d0c-2576e6a2c115 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:56:47.838517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0069ba71-a8d0-4937-9d0c-2576e6a2c115 service swift] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:56:47.848144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1212/2437] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:56:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:56:57.819116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:56:57.844805 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:DELETE=3,keystone:SELECT=38,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:06.988604 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610e1b12-522a-4d9b-adcf-18e1741d15f9 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:06.989157 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610e1b12-522a-4d9b-adcf-18e1741d15f9 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:06.989157 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610e1b12-522a-4d9b-adcf-18e1741d15f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:07.008658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-610e1b12-522a-4d9b-adcf-18e1741d15f9 None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:07.009862 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-610e1b12-522a-4d9b-adcf-18e1741d15f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:07.017316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1226/2438] 10.222.0.65 () {58 vars in 967 bytes} [Wed Aug 24 03:57:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:57:07.030040 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0069ba71-a8d0-4937-9d0c-2576e6a2c115 service swift] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:07.041022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0069ba71-a8d0-4937-9d0c-2576e6a2c115 service swift] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:07.052315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac7183a-b57c-4b5f-8591-e05bd0792968 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:07.052947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac7183a-b57c-4b5f-8591-e05bd0792968 service swift] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:07.053252 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac7183a-b57c-4b5f-8591-e05bd0792968 service swift] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:07.053498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac7183a-b57c-4b5f-8591-e05bd0792968 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:07.062469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac7183a-b57c-4b5f-8591-e05bd0792968 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:07.064052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac7183a-b57c-4b5f-8591-e05bd0792968 service swift] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:07.072152 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1213/2439] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 03:57:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 03:57:17.003532 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:17.068818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=6 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:27.039086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610e1b12-522a-4d9b-adcf-18e1741d15f9 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:27.040490 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-610e1b12-522a-4d9b-adcf-18e1741d15f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:27.096818 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610e1b12-522a-4d9b-adcf-18e1741d15f9 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:27.100633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364a6565-da81-4ea8-aa61-851f51d4b2e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:27.101246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364a6565-da81-4ea8-aa61-851f51d4b2e7 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:27.101529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364a6565-da81-4ea8-aa61-851f51d4b2e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:27.101803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364a6565-da81-4ea8-aa61-851f51d4b2e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:27.103866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-364a6565-da81-4ea8-aa61-851f51d4b2e7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:27.166879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364a6565-da81-4ea8-aa61-851f51d4b2e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:27.168339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364a6565-da81-4ea8-aa61-851f51d4b2e7 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:27.178283 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1227/2440] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:27.295227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac7183a-b57c-4b5f-8591-e05bd0792968 service swift] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:27.296721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1ac7183a-b57c-4b5f-8591-e05bd0792968 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:27.309706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac7183a-b57c-4b5f-8591-e05bd0792968 service swift] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:27.313274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417438d2-6d85-43d7-ab24-c264cd423366 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:27.314688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417438d2-6d85-43d7-ab24-c264cd423366 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:27.314963 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417438d2-6d85-43d7-ab24-c264cd423366 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:27.315261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417438d2-6d85-43d7-ab24-c264cd423366 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:27.316986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-417438d2-6d85-43d7-ab24-c264cd423366 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:27.328741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417438d2-6d85-43d7-ab24-c264cd423366 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:27.331096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417438d2-6d85-43d7-ab24-c264cd423366 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:27.341139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1214/2441] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:31.255110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364a6565-da81-4ea8-aa61-851f51d4b2e7 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:31.257255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-364a6565-da81-4ea8-aa61-851f51d4b2e7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:31.372071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364a6565-da81-4ea8-aa61-851f51d4b2e7 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:31.379035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f35d3b2-785e-4b2f-a855-adbdbc124869 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:31.380105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f35d3b2-785e-4b2f-a855-adbdbc124869 service placement] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:31.380578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f35d3b2-785e-4b2f-a855-adbdbc124869 service placement] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:31.381019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f35d3b2-785e-4b2f-a855-adbdbc124869 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:31.386917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4f35d3b2-785e-4b2f-a855-adbdbc124869 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:31.387419 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417438d2-6d85-43d7-ab24-c264cd423366 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:31.390236 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-417438d2-6d85-43d7-ab24-c264cd423366 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:31.402604 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f35d3b2-785e-4b2f-a855-adbdbc124869 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:31.407826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f35d3b2-785e-4b2f-a855-adbdbc124869 service placement] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:31.422032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1228/2442] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:31.423274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417438d2-6d85-43d7-ab24-c264cd423366 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:31.433095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:31.435213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:31.435675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:31.436348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] PATH_INFO: `/v3/users/77b4b56f008c44c081130377cf8ce630` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:31.438824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] RBAC: Authorizing `identity:delete_user(user_id=77b4b56f008c44c081130377cf8ce630)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:31.442182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:31.646020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77b4b56f008c44c081130377cf8ce630'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:31.646889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77b4b56f008c44c081130377cf8ce630'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:31.664643 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77b4b56f008c44c081130377cf8ce630'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:31.682040 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77b4b56f008c44c081130377cf8ce630'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:31.841999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77b4b56f008c44c081130377cf8ce630'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:31.851513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1215/2443] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:57:31 2022] DELETE /identity/v3/users/77b4b56f008c44c081130377cf8ce630 => generated 0 bytes in 465 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:31.855703 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f35d3b2-785e-4b2f-a855-adbdbc124869 service placement] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:31.857307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4f35d3b2-785e-4b2f-a855-adbdbc124869 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:31.871497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f35d3b2-785e-4b2f-a855-adbdbc124869 service placement] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:31.913913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:31.932942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d740d58-c806-45de-8809-ac1f52c503d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:31.933689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d740d58-c806-45de-8809-ac1f52c503d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:31.934041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d740d58-c806-45de-8809-ac1f52c503d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:31.934348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d740d58-c806-45de-8809-ac1f52c503d5 admin admin] PATH_INFO: `/v3/users/7f4991f9c6ca4fc99f756f75f9460808` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:31.935515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d740d58-c806-45de-8809-ac1f52c503d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f4991f9c6ca4fc99f756f75f9460808)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:31.937445 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d740d58-c806-45de-8809-ac1f52c503d5 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:32.129221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:32.131069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:32.145332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffbb899-3928-4801-ad1d-dd5c97f8a86f admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:32.150506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6473f362-34ac-4307-9fc0-5ca92343224e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:32.151690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6473f362-34ac-4307-9fc0-5ca92343224e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:32.152178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6473f362-34ac-4307-9fc0-5ca92343224e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:32.152457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6473f362-34ac-4307-9fc0-5ca92343224e admin admin] PATH_INFO: `/v3/projects/8f7520af1ae14ad28eec126353eaac53` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:32.153679 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6473f362-34ac-4307-9fc0-5ca92343224e admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f7520af1ae14ad28eec126353eaac53)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:32.164338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6473f362-34ac-4307-9fc0-5ca92343224e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:32.244567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3d740d58-c806-45de-8809-ac1f52c503d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f4991f9c6ca4fc99f756f75f9460808'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.245212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3d740d58-c806-45de-8809-ac1f52c503d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f4991f9c6ca4fc99f756f75f9460808'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.252268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3d740d58-c806-45de-8809-ac1f52c503d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f4991f9c6ca4fc99f756f75f9460808'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.252981 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3d740d58-c806-45de-8809-ac1f52c503d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f4991f9c6ca4fc99f756f75f9460808'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.304783 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6473f362-34ac-4307-9fc0-5ca92343224e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f7520af1ae14ad28eec126353eaac53', 'request_id': 'req-6473f362-34ac-4307-9fc0-5ca92343224e'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.310144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6473f362-34ac-4307-9fc0-5ca92343224e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f7520af1ae14ad28eec126353eaac53', 'request_id': 'req-6473f362-34ac-4307-9fc0-5ca92343224e'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.316380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6473f362-34ac-4307-9fc0-5ca92343224e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f7520af1ae14ad28eec126353eaac53', 'request_id': 'req-6473f362-34ac-4307-9fc0-5ca92343224e'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.321374 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-3d740d58-c806-45de-8809-ac1f52c503d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f4991f9c6ca4fc99f756f75f9460808'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.339602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1229/2444] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:57:31 2022] DELETE /identity/v3/users/7f4991f9c6ca4fc99f756f75f9460808 => generated 0 bytes in 485 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:57:32.422091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6473f362-34ac-4307-9fc0-5ca92343224e admin admin] The token cache is being invalidate because project 8f7520af1ae14ad28eec126353eaac53 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:57:32.423444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6473f362-34ac-4307-9fc0-5ca92343224e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.423646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-6473f362-34ac-4307-9fc0-5ca92343224e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.426680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1216/2445] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:57:32 2022] DELETE /identity/v3/projects/8f7520af1ae14ad28eec126353eaac53 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:57:32.431717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b644baa-69c6-4b54-8db8-e6c1e0630671 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:32.432055 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b644baa-69c6-4b54-8db8-e6c1e0630671 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:32.432334 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b644baa-69c6-4b54-8db8-e6c1e0630671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:32.453248 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5b644baa-69c6-4b54-8db8-e6c1e0630671 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:32.454522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5b644baa-69c6-4b54-8db8-e6c1e0630671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:32.519102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1230/2446] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:57:32.526790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c0b8f6-6208-4afa-bc9e-13c58471794f None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:32.527076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c0b8f6-6208-4afa-bc9e-13c58471794f None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:32.530495 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c0b8f6-6208-4afa-bc9e-13c58471794f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:32.547536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-06c0b8f6-6208-4afa-bc9e-13c58471794f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:32.549421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-06c0b8f6-6208-4afa-bc9e-13c58471794f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:32.557428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1217/2447] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:57:32.561631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b644baa-69c6-4b54-8db8-e6c1e0630671 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:32.571688 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b644baa-69c6-4b54-8db8-e6c1e0630671 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:32.576291 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8725da6-a585-49a6-acd5-8f586cc59ea8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:32.576859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8725da6-a585-49a6-acd5-8f586cc59ea8 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:32.577133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8725da6-a585-49a6-acd5-8f586cc59ea8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:32.577401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8725da6-a585-49a6-acd5-8f586cc59ea8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:32.578058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8725da6-a585-49a6-acd5-8f586cc59ea8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:57:32.578365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8725da6-a585-49a6-acd5-8f586cc59ea8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:32.580244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8725da6-a585-49a6-acd5-8f586cc59ea8 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:32.591559 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1231/2448] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:57:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:57:32.604807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c0b8f6-6208-4afa-bc9e-13c58471794f None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:32.615195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c0b8f6-6208-4afa-bc9e-13c58471794f None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:32.620207 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1a7bb5-1d7e-4b08-b4c9-8fc2568a36b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:32.620793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1a7bb5-1d7e-4b08-b4c9-8fc2568a36b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:32.621066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1a7bb5-1d7e-4b08-b4c9-8fc2568a36b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:32.621336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1a7bb5-1d7e-4b08-b4c9-8fc2568a36b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:32.621964 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1a7bb5-1d7e-4b08-b4c9-8fc2568a36b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:32.623973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1a7bb5-1d7e-4b08-b4c9-8fc2568a36b5 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:32.637491 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8725da6-a585-49a6-acd5-8f586cc59ea8 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:32.639377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e8725da6-a585-49a6-acd5-8f586cc59ea8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:32.655748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8725da6-a585-49a6-acd5-8f586cc59ea8 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:32.662050 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b695d9-acba-4cfe-ac75-af3646540c85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:32.663193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b695d9-acba-4cfe-ac75-af3646540c85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:32.663521 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b695d9-acba-4cfe-ac75-af3646540c85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:32.663810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b695d9-acba-4cfe-ac75-af3646540c85 admin admin] PATH_INFO: `/v3/projects/dbaf4ab3fd764ce88e7af0dbc3e6601e` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:32.664951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b695d9-acba-4cfe-ac75-af3646540c85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dbaf4ab3fd764ce88e7af0dbc3e6601e)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:32.666919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b695d9-acba-4cfe-ac75-af3646540c85 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:32.728947 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1218/2449] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:57:32 2022] POST /identity/v3/projects => generated 375 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:57:32.733085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1a7bb5-1d7e-4b08-b4c9-8fc2568a36b5 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:32.749344 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=80,keystone:DELETE=17,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:32.805276 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1a7bb5-1d7e-4b08-b4c9-8fc2568a36b5 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:32.810118 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f61cc60-8aec-48e1-80a0-2f4928fac8a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:32.810754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f61cc60-8aec-48e1-80a0-2f4928fac8a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:32.811026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f61cc60-8aec-48e1-80a0-2f4928fac8a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:32.811296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f61cc60-8aec-48e1-80a0-2f4928fac8a3 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:32.812019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f61cc60-8aec-48e1-80a0-2f4928fac8a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:32.814928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f61cc60-8aec-48e1-80a0-2f4928fac8a3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:32.859108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:32.866351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b7b695d9-acba-4cfe-ac75-af3646540c85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dbaf4ab3fd764ce88e7af0dbc3e6601e', 'request_id': 'req-b7b695d9-acba-4cfe-ac75-af3646540c85'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.871989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b7b695d9-acba-4cfe-ac75-af3646540c85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dbaf4ab3fd764ce88e7af0dbc3e6601e', 'request_id': 'req-b7b695d9-acba-4cfe-ac75-af3646540c85'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.928662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1219/2450] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:57:32 2022] POST /identity/v3/users => generated 288 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:57:32.932519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f61cc60-8aec-48e1-80a0-2f4928fac8a3 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:32.942758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f61cc60-8aec-48e1-80a0-2f4928fac8a3 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:32.950218 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b7b695d9-acba-4cfe-ac75-af3646540c85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dbaf4ab3fd764ce88e7af0dbc3e6601e', 'request_id': 'req-b7b695d9-acba-4cfe-ac75-af3646540c85'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.962651 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b7b695d9-acba-4cfe-ac75-af3646540c85 admin admin] The token cache is being invalidate because project dbaf4ab3fd764ce88e7af0dbc3e6601e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:57:32.962933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b7b695d9-acba-4cfe-ac75-af3646540c85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.963600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b7b695d9-acba-4cfe-ac75-af3646540c85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:32.965087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1232/2451] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:57:32 2022] DELETE /identity/v3/projects/dbaf4ab3fd764ce88e7af0dbc3e6601e => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:57:32.969977 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1830ded9-d0d7-448d-a094-f707e3a28ebf None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:32.970290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1830ded9-d0d7-448d-a094-f707e3a28ebf None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:32.970558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1830ded9-d0d7-448d-a094-f707e3a28ebf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:32.990640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1830ded9-d0d7-448d-a094-f707e3a28ebf None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:32.996548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1830ded9-d0d7-448d-a094-f707e3a28ebf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:32.999320 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7128e8db-fd97-4392-b5ce-15ea22c4bdf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:32.999922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7128e8db-fd97-4392-b5ce-15ea22c4bdf0 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.000193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7128e8db-fd97-4392-b5ce-15ea22c4bdf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.000459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7128e8db-fd97-4392-b5ce-15ea22c4bdf0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.001836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7128e8db-fd97-4392-b5ce-15ea22c4bdf0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.003451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7128e8db-fd97-4392-b5ce-15ea22c4bdf0 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.013073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1220/2452] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:32 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:57:33.017683 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7128e8db-fd97-4392-b5ce-15ea22c4bdf0 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.019491 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7128e8db-fd97-4392-b5ce-15ea22c4bdf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:33.031803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7128e8db-fd97-4392-b5ce-15ea22c4bdf0 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:33.037084 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980c9717-170f-4a84-862e-9c0a5fa7ed94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:33.037669 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980c9717-170f-4a84-862e-9c0a5fa7ed94 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.037944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980c9717-170f-4a84-862e-9c0a5fa7ed94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.038238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980c9717-170f-4a84-862e-9c0a5fa7ed94 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.038870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980c9717-170f-4a84-862e-9c0a5fa7ed94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.040432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980c9717-170f-4a84-862e-9c0a5fa7ed94 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.052331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1233/2453] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:57:33.053891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1221/2454] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:33 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:57:33.056589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1830ded9-d0d7-448d-a094-f707e3a28ebf None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.061115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cccba4d-6b24-43bc-8017-51fcfab25df1 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.061405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cccba4d-6b24-43bc-8017-51fcfab25df1 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.061705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cccba4d-6b24-43bc-8017-51fcfab25df1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.066349 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1830ded9-d0d7-448d-a094-f707e3a28ebf None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:33.071117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326b54e3-8416-4f85-902f-0b430dd7f5dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:33.071763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326b54e3-8416-4f85-902f-0b430dd7f5dd admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.072041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326b54e3-8416-4f85-902f-0b430dd7f5dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.072312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326b54e3-8416-4f85-902f-0b430dd7f5dd admin admin] PATH_INFO: `/v3/projects/53090206d87c4d9abbfed95ca4832a67/users/107b596147ea42968db0be5330080a01/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.081875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-5cccba4d-6b24-43bc-8017-51fcfab25df1 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:33.083180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5cccba4d-6b24-43bc-8017-51fcfab25df1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:33.088165 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326b54e3-8416-4f85-902f-0b430dd7f5dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=53090206d87c4d9abbfed95ca4832a67, user_id=107b596147ea42968db0be5330080a01, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.090912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326b54e3-8416-4f85-902f-0b430dd7f5dd admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.091293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1222/2455] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:57:33.096367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cccba4d-6b24-43bc-8017-51fcfab25df1 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.105862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cccba4d-6b24-43bc-8017-51fcfab25df1 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:33.114671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2bbb92-70b6-49f7-88f7-214f4c9aadeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:33.114671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2bbb92-70b6-49f7-88f7-214f4c9aadeb admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.114671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2bbb92-70b6-49f7-88f7-214f4c9aadeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.114671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2bbb92-70b6-49f7-88f7-214f4c9aadeb admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.115095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2bbb92-70b6-49f7-88f7-214f4c9aadeb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:57:33.115095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2bbb92-70b6-49f7-88f7-214f4c9aadeb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.115095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2bbb92-70b6-49f7-88f7-214f4c9aadeb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.126462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1223/2456] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:57:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:57:33.130888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2bbb92-70b6-49f7-88f7-214f4c9aadeb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.140931 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2bbb92-70b6-49f7-88f7-214f4c9aadeb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:33.146358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62d75a3-06ae-4f1f-a11e-d96c4be35f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:33.149245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62d75a3-06ae-4f1f-a11e-d96c4be35f56 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.149245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62d75a3-06ae-4f1f-a11e-d96c4be35f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.149245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62d75a3-06ae-4f1f-a11e-d96c4be35f56 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.149245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62d75a3-06ae-4f1f-a11e-d96c4be35f56 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.150031 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62d75a3-06ae-4f1f-a11e-d96c4be35f56 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.195034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1234/2457] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:33 2022] PUT /identity/v3/projects/53090206d87c4d9abbfed95ca4832a67/users/107b596147ea42968db0be5330080a01/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:57:33.196789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfef29c-e01f-49d5-9329-02e13820d42f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.197077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfef29c-e01f-49d5-9329-02e13820d42f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.197351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfef29c-e01f-49d5-9329-02e13820d42f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.225457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-fdfef29c-e01f-49d5-9329-02e13820d42f None None] MFA Rules not processed for user `107b596147ea42968db0be5330080a01`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:33.226727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-fdfef29c-e01f-49d5-9329-02e13820d42f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:33.290596 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1235/2458] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:33 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:57:33.295088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfef29c-e01f-49d5-9329-02e13820d42f None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.358064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfef29c-e01f-49d5-9329-02e13820d42f None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:33.363827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896052f6-5565-44a8-b9f7-3caf8c9e07f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:33.364050 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1224/2459] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:57:33 2022] POST /identity/v3/projects => generated 365 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:57:33.364432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896052f6-5565-44a8-b9f7-3caf8c9e07f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.364718 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896052f6-5565-44a8-b9f7-3caf8c9e07f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.364986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896052f6-5565-44a8-b9f7-3caf8c9e07f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.365631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896052f6-5565-44a8-b9f7-3caf8c9e07f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.367605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896052f6-5565-44a8-b9f7-3caf8c9e07f4 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.368146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62d75a3-06ae-4f1f-a11e-d96c4be35f56 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.418705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:33.433307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62d75a3-06ae-4f1f-a11e-d96c4be35f56 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:33.447096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0bc11d-6d9a-4ebb-99cc-f6a98db13999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:33.447096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0bc11d-6d9a-4ebb-99cc-f6a98db13999 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.447096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0bc11d-6d9a-4ebb-99cc-f6a98db13999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.447096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0bc11d-6d9a-4ebb-99cc-f6a98db13999 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.447517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0bc11d-6d9a-4ebb-99cc-f6a98db13999 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.447517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0bc11d-6d9a-4ebb-99cc-f6a98db13999 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.476730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:33.529854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1236/2460] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:57:33 2022] POST /identity/v3/projects => generated 377 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:57:33.533488 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896052f6-5565-44a8-b9f7-3caf8c9e07f4 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.586925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1225/2461] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:57:33 2022] POST /identity/v3/users => generated 283 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:57:33.595937 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896052f6-5565-44a8-b9f7-3caf8c9e07f4 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:33.598783 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0bc11d-6d9a-4ebb-99cc-f6a98db13999 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.599576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4e30ca-f7fa-44c0-9f1b-4a84040b39f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:33.600179 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4e30ca-f7fa-44c0-9f1b-4a84040b39f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.600456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4e30ca-f7fa-44c0-9f1b-4a84040b39f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.600727 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4e30ca-f7fa-44c0-9f1b-4a84040b39f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.601341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4e30ca-f7fa-44c0-9f1b-4a84040b39f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.603551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4e30ca-f7fa-44c0-9f1b-4a84040b39f1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.609040 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0bc11d-6d9a-4ebb-99cc-f6a98db13999 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:33.614833 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b69071-b45a-4cb6-979a-572f6227430e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:33.615527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b69071-b45a-4cb6-979a-572f6227430e admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.615807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b69071-b45a-4cb6-979a-572f6227430e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.616102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b69071-b45a-4cb6-979a-572f6227430e admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.616732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b69071-b45a-4cb6-979a-572f6227430e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.618392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b69071-b45a-4cb6-979a-572f6227430e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.629274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1226/2462] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:57:33.633160 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b69071-b45a-4cb6-979a-572f6227430e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.642864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b69071-b45a-4cb6-979a-572f6227430e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:33.648202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f4ee53-1b86-4704-8779-b3a6778971d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:33.648763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f4ee53-1b86-4704-8779-b3a6778971d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.649039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f4ee53-1b86-4704-8779-b3a6778971d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.649308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f4ee53-1b86-4704-8779-b3a6778971d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.649924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f4ee53-1b86-4704-8779-b3a6778971d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.651508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f4ee53-1b86-4704-8779-b3a6778971d9 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.662093 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1227/2463] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:57:33.665980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f4ee53-1b86-4704-8779-b3a6778971d9 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.675957 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f4ee53-1b86-4704-8779-b3a6778971d9 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:33.680492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360c8e2a-e04d-41b6-9a42-f4d604cfe224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:33.681100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360c8e2a-e04d-41b6-9a42-f4d604cfe224 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.681376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360c8e2a-e04d-41b6-9a42-f4d604cfe224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.681649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360c8e2a-e04d-41b6-9a42-f4d604cfe224 admin admin] PATH_INFO: `/v3/projects/981b6a68edd2494cb81bb5ad9f6ce0a6/users/530cc55346374cffbdf8bf4bf6f09069/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.696500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360c8e2a-e04d-41b6-9a42-f4d604cfe224 admin admin] RBAC: Authorizing `identity:create_grant(project_id=981b6a68edd2494cb81bb5ad9f6ce0a6, user_id=530cc55346374cffbdf8bf4bf6f09069, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.699315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360c8e2a-e04d-41b6-9a42-f4d604cfe224 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.757313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1237/2464] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:57:33 2022] POST /identity/v3/users => generated 295 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:57:33.761239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4e30ca-f7fa-44c0-9f1b-4a84040b39f1 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.771124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4e30ca-f7fa-44c0-9f1b-4a84040b39f1 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:33.775976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f85c075-6725-4f10-8afe-4a19b727892a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:33.776576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f85c075-6725-4f10-8afe-4a19b727892a admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.776855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f85c075-6725-4f10-8afe-4a19b727892a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.777128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f85c075-6725-4f10-8afe-4a19b727892a admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.777807 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f85c075-6725-4f10-8afe-4a19b727892a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.779419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f85c075-6725-4f10-8afe-4a19b727892a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.790035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1238/2465] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:57:33.799494 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f85c075-6725-4f10-8afe-4a19b727892a admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.809402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f85c075-6725-4f10-8afe-4a19b727892a admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:33.814753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf33320-0a20-4ddd-a086-7b5a083b9d60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:33.815407 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf33320-0a20-4ddd-a086-7b5a083b9d60 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.815687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf33320-0a20-4ddd-a086-7b5a083b9d60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.815961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf33320-0a20-4ddd-a086-7b5a083b9d60 admin admin] PATH_INFO: `/v3/projects/ac092cb6c213427e867a48c96bb68acb/users/9b6f91894b1c45379b8b39c80d21dd75/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.830380 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf33320-0a20-4ddd-a086-7b5a083b9d60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac092cb6c213427e867a48c96bb68acb, user_id=9b6f91894b1c45379b8b39c80d21dd75, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:33.833021 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf33320-0a20-4ddd-a086-7b5a083b9d60 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:33.910527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1228/2466] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:33 2022] PUT /identity/v3/projects/981b6a68edd2494cb81bb5ad9f6ce0a6/users/530cc55346374cffbdf8bf4bf6f09069/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:57:33.921042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae548b5d-2245-4922-8b50-046a70f0f2ed None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:33.921329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae548b5d-2245-4922-8b50-046a70f0f2ed None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:33.921603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae548b5d-2245-4922-8b50-046a70f0f2ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:33.933013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1239/2467] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:33 2022] PUT /identity/v3/projects/ac092cb6c213427e867a48c96bb68acb/users/9b6f91894b1c45379b8b39c80d21dd75/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:57:33.937443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf33320-0a20-4ddd-a086-7b5a083b9d60 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:33.952170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-ae548b5d-2245-4922-8b50-046a70f0f2ed None None] MFA Rules not processed for user `530cc55346374cffbdf8bf4bf6f09069`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:33.953368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ae548b5d-2245-4922-8b50-046a70f0f2ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:34.003103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf33320-0a20-4ddd-a086-7b5a083b9d60 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:34.007873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da2b7b3-e221-4d47-8661-2fe201d4bfca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:34.008475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da2b7b3-e221-4d47-8661-2fe201d4bfca admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:34.008755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da2b7b3-e221-4d47-8661-2fe201d4bfca admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:34.009027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da2b7b3-e221-4d47-8661-2fe201d4bfca admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:34.009666 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da2b7b3-e221-4d47-8661-2fe201d4bfca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:34.011291 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da2b7b3-e221-4d47-8661-2fe201d4bfca admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:34.020425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1229/2468] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:33 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:57:34.024926 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1240/2469] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:33 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:57:34.025065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae548b5d-2245-4922-8b50-046a70f0f2ed None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:34.025873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da2b7b3-e221-4d47-8661-2fe201d4bfca admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:34.035839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae548b5d-2245-4922-8b50-046a70f0f2ed None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:34.035997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da2b7b3-e221-4d47-8661-2fe201d4bfca admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:34.040384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ef8db6-f6ed-404d-a92c-c9e24a046bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:34.040576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502e3822-25c9-4e2f-8830-d6975d10a5b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:34.040985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ef8db6-f6ed-404d-a92c-c9e24a046bc3 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:34.041216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502e3822-25c9-4e2f-8830-d6975d10a5b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:34.041316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ef8db6-f6ed-404d-a92c-c9e24a046bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:34.041498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502e3822-25c9-4e2f-8830-d6975d10a5b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:34.041604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ef8db6-f6ed-404d-a92c-c9e24a046bc3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:34.041773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502e3822-25c9-4e2f-8830-d6975d10a5b3 admin admin] PATH_INFO: `/v3/projects/ac092cb6c213427e867a48c96bb68acb/users/9b6f91894b1c45379b8b39c80d21dd75/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:34.042134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ef8db6-f6ed-404d-a92c-c9e24a046bc3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:34.043749 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502e3822-25c9-4e2f-8830-d6975d10a5b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac092cb6c213427e867a48c96bb68acb, user_id=9b6f91894b1c45379b8b39c80d21dd75, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:34.044050 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ef8db6-f6ed-404d-a92c-c9e24a046bc3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:34.046390 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502e3822-25c9-4e2f-8830-d6975d10a5b3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:34.052048 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:34.057007 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1241/2470] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:34 2022] PUT /identity/v3/projects/ac092cb6c213427e867a48c96bb68acb/users/9b6f91894b1c45379b8b39c80d21dd75/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:34.071184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a55248-7caa-4ab4-8ca6-3872916b040f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:34.071184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a55248-7caa-4ab4-8ca6-3872916b040f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:34.071184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a55248-7caa-4ab4-8ca6-3872916b040f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:34.096815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-35a55248-7caa-4ab4-8ca6-3872916b040f None None] MFA Rules not processed for user `9b6f91894b1c45379b8b39c80d21dd75`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:34.098978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-35a55248-7caa-4ab4-8ca6-3872916b040f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:34.137260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1230/2471] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:57:34 2022] POST /identity/v3/projects => generated 365 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:57:34.138578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ef8db6-f6ed-404d-a92c-c9e24a046bc3 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:34.169282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1242/2472] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:34 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:57:34.180847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a55248-7caa-4ab4-8ca6-3872916b040f None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:34.182401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-35a55248-7caa-4ab4-8ca6-3872916b040f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:34.202789 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:34.207588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ef8db6-f6ed-404d-a92c-c9e24a046bc3 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:34.212645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f8c1ac-5336-4cb8-ab60-9d658d93d0d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:34.215786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f8c1ac-5336-4cb8-ab60-9d658d93d0d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:34.215786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f8c1ac-5336-4cb8-ab60-9d658d93d0d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:34.215786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f8c1ac-5336-4cb8-ab60-9d658d93d0d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:34.215786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f8c1ac-5336-4cb8-ab60-9d658d93d0d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:34.216322 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f8c1ac-5336-4cb8-ab60-9d658d93d0d8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:34.261699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a55248-7caa-4ab4-8ca6-3872916b040f None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:34.309614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fbdaf3-1516-49c5-9d63-305ae2d701d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:34.310242 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fbdaf3-1516-49c5-9d63-305ae2d701d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:34.310526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fbdaf3-1516-49c5-9d63-305ae2d701d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:34.310801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fbdaf3-1516-49c5-9d63-305ae2d701d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:34.322904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1231/2473] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:57:34 2022] POST /identity/v3/users => generated 289 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:57:34.326790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f8c1ac-5336-4cb8-ab60-9d658d93d0d8 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:34.337540 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f8c1ac-5336-4cb8-ab60-9d658d93d0d8 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:34.342623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d790f9-46af-4726-80db-f9d44d7b2a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:34.343256 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d790f9-46af-4726-80db-f9d44d7b2a63 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:34.343559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d790f9-46af-4726-80db-f9d44d7b2a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:34.343830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d790f9-46af-4726-80db-f9d44d7b2a63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:34.344708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d790f9-46af-4726-80db-f9d44d7b2a63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:34.345665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:34.346271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d790f9-46af-4726-80db-f9d44d7b2a63 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:34.357352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1232/2474] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:57:34.362175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d790f9-46af-4726-80db-f9d44d7b2a63 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:34.371184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fbdaf3-1516-49c5-9d63-305ae2d701d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:34.371921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d790f9-46af-4726-80db-f9d44d7b2a63 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:34.372752 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fbdaf3-1516-49c5-9d63-305ae2d701d5 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:34.377798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7366fd77-ad45-422b-91cf-7bdc0eb9dc2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:34.378478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7366fd77-ad45-422b-91cf-7bdc0eb9dc2c admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:34.378830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7366fd77-ad45-422b-91cf-7bdc0eb9dc2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:34.379105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7366fd77-ad45-422b-91cf-7bdc0eb9dc2c admin admin] PATH_INFO: `/v3/projects/88b3ade78a1f4df5a77700adf1e392ce/users/3ef475986a614ad783826e626abccf2f/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:34.382666 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1243/2475] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:34 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:57:34.394042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7366fd77-ad45-422b-91cf-7bdc0eb9dc2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=88b3ade78a1f4df5a77700adf1e392ce, user_id=3ef475986a614ad783826e626abccf2f, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:34.396722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7366fd77-ad45-422b-91cf-7bdc0eb9dc2c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:34.465352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1233/2476] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:34 2022] PUT /identity/v3/projects/88b3ade78a1f4df5a77700adf1e392ce/users/3ef475986a614ad783826e626abccf2f/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:57:34.469615 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fbdaf3-1516-49c5-9d63-305ae2d701d5 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:34.481944 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fbdaf3-1516-49c5-9d63-305ae2d701d5 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:34.541459 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee89fe45-ceb4-46c8-b0df-0663a6a4c44a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:34.542061 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee89fe45-ceb4-46c8-b0df-0663a6a4c44a admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:34.542370 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee89fe45-ceb4-46c8-b0df-0663a6a4c44a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:34.542642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee89fe45-ceb4-46c8-b0df-0663a6a4c44a admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:34.543349 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee89fe45-ceb4-46c8-b0df-0663a6a4c44a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:34.544957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee89fe45-ceb4-46c8-b0df-0663a6a4c44a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:34.555191 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1244/2477] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:34 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:57:34.559519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7366fd77-ad45-422b-91cf-7bdc0eb9dc2c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:34.570815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7366fd77-ad45-422b-91cf-7bdc0eb9dc2c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:34.576008 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35a7dbb-948c-43f5-ac34-458b885c0dc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:34.576844 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35a7dbb-948c-43f5-ac34-458b885c0dc7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:34.577127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35a7dbb-948c-43f5-ac34-458b885c0dc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:34.577574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35a7dbb-948c-43f5-ac34-458b885c0dc7 admin admin] PATH_INFO: `/v3/projects/88b3ade78a1f4df5a77700adf1e392ce/users/3ef475986a614ad783826e626abccf2f/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:34.579715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35a7dbb-948c-43f5-ac34-458b885c0dc7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=88b3ade78a1f4df5a77700adf1e392ce, user_id=3ef475986a614ad783826e626abccf2f, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:34.582451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35a7dbb-948c-43f5-ac34-458b885c0dc7 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:34.593199 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1234/2478] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:34 2022] PUT /identity/v3/projects/88b3ade78a1f4df5a77700adf1e392ce/users/3ef475986a614ad783826e626abccf2f/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:34.605522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced62c0a-d32d-4d4c-8868-f9e7136cfa46 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:34.605803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced62c0a-d32d-4d4c-8868-f9e7136cfa46 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:34.606083 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced62c0a-d32d-4d4c-8868-f9e7136cfa46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:34.635367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-ced62c0a-d32d-4d4c-8868-f9e7136cfa46 None None] MFA Rules not processed for user `3ef475986a614ad783826e626abccf2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:34.636619 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ced62c0a-d32d-4d4c-8868-f9e7136cfa46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:34.708726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1245/2479] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:34 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:34.720871 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35a7dbb-948c-43f5-ac34-458b885c0dc7 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:34.722653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f35a7dbb-948c-43f5-ac34-458b885c0dc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:34.783461 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35a7dbb-948c-43f5-ac34-458b885c0dc7 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:34.787227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce9f9d5-1936-468b-9f00-6b6d5f3cb403 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:34.787828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce9f9d5-1936-468b-9f00-6b6d5f3cb403 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:34.788101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce9f9d5-1936-468b-9f00-6b6d5f3cb403 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:34.788365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce9f9d5-1936-468b-9f00-6b6d5f3cb403 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:34.798494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce9f9d5-1936-468b-9f00-6b6d5f3cb403 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:34.800047 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce9f9d5-1936-468b-9f00-6b6d5f3cb403 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:34.809728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1235/2480] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:34 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:35.819292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced62c0a-d32d-4d4c-8868-f9e7136cfa46 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:35.877960 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:35.883463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced62c0a-d32d-4d4c-8868-f9e7136cfa46 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:35.888174 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:35.888836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:35.889115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:35.889389 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] PATH_INFO: `/v3/users/530cc55346374cffbdf8bf4bf6f09069` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:35.890555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=530cc55346374cffbdf8bf4bf6f09069)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:35.892650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:36.055042 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '530cc55346374cffbdf8bf4bf6f09069'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:36.055652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '530cc55346374cffbdf8bf4bf6f09069'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:36.062770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '530cc55346374cffbdf8bf4bf6f09069'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:36.064148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '530cc55346374cffbdf8bf4bf6f09069'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:36.121607 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '530cc55346374cffbdf8bf4bf6f09069'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:36.143058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1246/2481] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:57:35 2022] DELETE /identity/v3/users/530cc55346374cffbdf8bf4bf6f09069 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:57:36.147690 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff0fb86-f7c6-455a-93d5-1e7883c910f4 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:36.148166 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff0fb86-f7c6-455a-93d5-1e7883c910f4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:36.148271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff0fb86-f7c6-455a-93d5-1e7883c910f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:36.167687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-6ff0fb86-f7c6-455a-93d5-1e7883c910f4 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:36.168890 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6ff0fb86-f7c6-455a-93d5-1e7883c910f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:36.240581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1236/2482] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:36.250468 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:36.252998 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:36.315451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08138031-dc2f-4102-abad-c873e2e4bfd7 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:36.319102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6003ead2-dbf8-4180-92dd-6748bd1c7f64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:36.319736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6003ead2-dbf8-4180-92dd-6748bd1c7f64 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:36.320017 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6003ead2-dbf8-4180-92dd-6748bd1c7f64 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:36.320289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6003ead2-dbf8-4180-92dd-6748bd1c7f64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:36.330343 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6003ead2-dbf8-4180-92dd-6748bd1c7f64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:36.331817 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6003ead2-dbf8-4180-92dd-6748bd1c7f64 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:36.342201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1247/2483] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:36.798297 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff0fb86-f7c6-455a-93d5-1e7883c910f4 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:36.799862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6ff0fb86-f7c6-455a-93d5-1e7883c910f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:36.815146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff0fb86-f7c6-455a-93d5-1e7883c910f4 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:36.823648 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:36.823648 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:36.823648 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:36.823648 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] PATH_INFO: `/v3/projects/981b6a68edd2494cb81bb5ad9f6ce0a6` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:36.824469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=981b6a68edd2494cb81bb5ad9f6ce0a6)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:36.825115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:36.981257 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:36.990756 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '981b6a68edd2494cb81bb5ad9f6ce0a6', 'request_id': 'req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:36.996120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '981b6a68edd2494cb81bb5ad9f6ce0a6', 'request_id': 'req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:37.002192 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '981b6a68edd2494cb81bb5ad9f6ce0a6', 'request_id': 'req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:37.092299 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] The token cache is being invalidate because project 981b6a68edd2494cb81bb5ad9f6ce0a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:57:37.097646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:37.097646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:37.097646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1237/2484] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:57:36 2022] DELETE /identity/v3/projects/981b6a68edd2494cb81bb5ad9f6ce0a6 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:57:37.099852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6003ead2-dbf8-4180-92dd-6748bd1c7f64 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:37.109913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6003ead2-dbf8-4180-92dd-6748bd1c7f64 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:37.167088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b9def0-bc9e-40b8-b25c-f6d83bdf9cd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:37.167712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b9def0-bc9e-40b8-b25c-f6d83bdf9cd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:37.167991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b9def0-bc9e-40b8-b25c-f6d83bdf9cd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:37.168262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b9def0-bc9e-40b8-b25c-f6d83bdf9cd0 admin admin] PATH_INFO: `/v3/users/3ef475986a614ad783826e626abccf2f` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:37.169397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b9def0-bc9e-40b8-b25c-f6d83bdf9cd0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ef475986a614ad783826e626abccf2f)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:37.171817 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b9def0-bc9e-40b8-b25c-f6d83bdf9cd0 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:37.193597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:37.352960 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a0b9def0-bc9e-40b8-b25c-f6d83bdf9cd0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ef475986a614ad783826e626abccf2f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:37.353899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a0b9def0-bc9e-40b8-b25c-f6d83bdf9cd0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ef475986a614ad783826e626abccf2f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:37.360892 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a0b9def0-bc9e-40b8-b25c-f6d83bdf9cd0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ef475986a614ad783826e626abccf2f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:37.361516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a0b9def0-bc9e-40b8-b25c-f6d83bdf9cd0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ef475986a614ad783826e626abccf2f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:37.441805 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a0b9def0-bc9e-40b8-b25c-f6d83bdf9cd0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ef475986a614ad783826e626abccf2f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:37.459858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1248/2485] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:57:37 2022] DELETE /identity/v3/users/3ef475986a614ad783826e626abccf2f => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:37.950735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:37.952318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:37.969135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c0ff53-88ee-41ea-9c7b-f592ca75e5cb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:38.025806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6219aea-d83a-4467-b6e1-45a9bd8699a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:38.026533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6219aea-d83a-4467-b6e1-45a9bd8699a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:38.026816 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6219aea-d83a-4467-b6e1-45a9bd8699a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:38.027119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6219aea-d83a-4467-b6e1-45a9bd8699a8 admin admin] PATH_INFO: `/v3/projects/88b3ade78a1f4df5a77700adf1e392ce` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:38.028238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6219aea-d83a-4467-b6e1-45a9bd8699a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=88b3ade78a1f4df5a77700adf1e392ce)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:38.030169 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6219aea-d83a-4467-b6e1-45a9bd8699a8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:38.167066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a6219aea-d83a-4467-b6e1-45a9bd8699a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88b3ade78a1f4df5a77700adf1e392ce', 'request_id': 'req-a6219aea-d83a-4467-b6e1-45a9bd8699a8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:38.173352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a6219aea-d83a-4467-b6e1-45a9bd8699a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88b3ade78a1f4df5a77700adf1e392ce', 'request_id': 'req-a6219aea-d83a-4467-b6e1-45a9bd8699a8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:38.180488 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a6219aea-d83a-4467-b6e1-45a9bd8699a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88b3ade78a1f4df5a77700adf1e392ce', 'request_id': 'req-a6219aea-d83a-4467-b6e1-45a9bd8699a8'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:38.286462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a6219aea-d83a-4467-b6e1-45a9bd8699a8 admin admin] The token cache is being invalidate because project 88b3ade78a1f4df5a77700adf1e392ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:57:38.286686 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a6219aea-d83a-4467-b6e1-45a9bd8699a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:38.287285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a6219aea-d83a-4467-b6e1-45a9bd8699a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:38.288812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1238/2486] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:57:37 2022] DELETE /identity/v3/projects/88b3ade78a1f4df5a77700adf1e392ce => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:38.295035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e639c4-d20d-401f-b8a5-37b054eea453 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:38.295571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e639c4-d20d-401f-b8a5-37b054eea453 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:38.295571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e639c4-d20d-401f-b8a5-37b054eea453 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:38.315122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-45e639c4-d20d-401f-b8a5-37b054eea453 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:38.329482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-45e639c4-d20d-401f-b8a5-37b054eea453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:38.377377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1249/2487] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:38.385248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116e569e-db74-495a-a60b-1ac44b37df12 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:38.385561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116e569e-db74-495a-a60b-1ac44b37df12 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:38.385841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116e569e-db74-495a-a60b-1ac44b37df12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:38.406464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-116e569e-db74-495a-a60b-1ac44b37df12 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:38.408353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-116e569e-db74-495a-a60b-1ac44b37df12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:38.418755 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1239/2488] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:38.422956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e639c4-d20d-401f-b8a5-37b054eea453 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:38.433035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e639c4-d20d-401f-b8a5-37b054eea453 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:38.439152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ada297-0907-4924-ac26-14d52eb65e1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:38.439763 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ada297-0907-4924-ac26-14d52eb65e1d admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:38.440047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ada297-0907-4924-ac26-14d52eb65e1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:38.440319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ada297-0907-4924-ac26-14d52eb65e1d admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:38.440973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ada297-0907-4924-ac26-14d52eb65e1d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:57:38.441275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ada297-0907-4924-ac26-14d52eb65e1d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:38.442931 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ada297-0907-4924-ac26-14d52eb65e1d admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:38.455677 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1250/2489] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:57:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:57:38.466431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116e569e-db74-495a-a60b-1ac44b37df12 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:38.475850 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116e569e-db74-495a-a60b-1ac44b37df12 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:38.480847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7e0273-5438-4e04-9419-ff362af081bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:38.481443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7e0273-5438-4e04-9419-ff362af081bd admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:38.481722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7e0273-5438-4e04-9419-ff362af081bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:38.481992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7e0273-5438-4e04-9419-ff362af081bd admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:38.482654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7e0273-5438-4e04-9419-ff362af081bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:38.484538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7e0273-5438-4e04-9419-ff362af081bd admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:38.569193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1240/2490] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:57:38 2022] POST /identity/v3/projects => generated 387 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:57:38.573407 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ada297-0907-4924-ac26-14d52eb65e1d admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:38.584323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ada297-0907-4924-ac26-14d52eb65e1d admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:38.644370 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f25962-3d76-4c75-add2-e838fd98d907 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:38.644963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f25962-3d76-4c75-add2-e838fd98d907 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:38.645240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f25962-3d76-4c75-add2-e838fd98d907 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:38.645508 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f25962-3d76-4c75-add2-e838fd98d907 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:38.646178 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f25962-3d76-4c75-add2-e838fd98d907 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:38.652053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f25962-3d76-4c75-add2-e838fd98d907 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:38.786046 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1251/2491] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:57:38 2022] POST /identity/v3/users => generated 294 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:57:38.790309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7e0273-5438-4e04-9419-ff362af081bd admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:38.801496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7e0273-5438-4e04-9419-ff362af081bd admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:38.806043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f08a85-653c-4406-9aac-ef30daa65a34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:38.811560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f08a85-653c-4406-9aac-ef30daa65a34 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:38.811560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f08a85-653c-4406-9aac-ef30daa65a34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:38.811560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f08a85-653c-4406-9aac-ef30daa65a34 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:38.811560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f08a85-653c-4406-9aac-ef30daa65a34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:38.811560 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f08a85-653c-4406-9aac-ef30daa65a34 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:38.820093 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1241/2492] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:38.823988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f25962-3d76-4c75-add2-e838fd98d907 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:38.835019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f25962-3d76-4c75-add2-e838fd98d907 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:38.840238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e09672-81e8-4232-a18d-242fe0137cc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:38.840238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e09672-81e8-4232-a18d-242fe0137cc9 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:38.840577 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e09672-81e8-4232-a18d-242fe0137cc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:38.840848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e09672-81e8-4232-a18d-242fe0137cc9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:38.841498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e09672-81e8-4232-a18d-242fe0137cc9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:38.843256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e09672-81e8-4232-a18d-242fe0137cc9 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:38.854138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1252/2493] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:38 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:38.858356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f08a85-653c-4406-9aac-ef30daa65a34 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:38.868971 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f08a85-653c-4406-9aac-ef30daa65a34 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:38.873492 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91296658-2535-4c0b-b99a-2182be14317c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:38.874841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91296658-2535-4c0b-b99a-2182be14317c admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:38.874841 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91296658-2535-4c0b-b99a-2182be14317c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:38.875049 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91296658-2535-4c0b-b99a-2182be14317c admin admin] PATH_INFO: `/v3/projects/6bdd915634de4482b0f7022dd0d3e0de/users/18985578c86f442997471fea0df42e15/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:38.887481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:38.890593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91296658-2535-4c0b-b99a-2182be14317c admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bdd915634de4482b0f7022dd0d3e0de, user_id=18985578c86f442997471fea0df42e15, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:38.893440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91296658-2535-4c0b-b99a-2182be14317c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:38.973369 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1242/2494] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:38 2022] PUT /identity/v3/projects/6bdd915634de4482b0f7022dd0d3e0de/users/18985578c86f442997471fea0df42e15/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:38.984018 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f407fed5-c82d-4cb0-a462-101cc70ed133 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:38.984392 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f407fed5-c82d-4cb0-a462-101cc70ed133 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:38.984667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f407fed5-c82d-4cb0-a462-101cc70ed133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:38.992788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:DELETE=8,keystone:SELECT=88,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:39.013910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-f407fed5-c82d-4cb0-a462-101cc70ed133 None None] MFA Rules not processed for user `18985578c86f442997471fea0df42e15`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:39.015162 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f407fed5-c82d-4cb0-a462-101cc70ed133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:39.080116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1253/2495] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:38 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:39.084573 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91296658-2535-4c0b-b99a-2182be14317c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:39.147347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91296658-2535-4c0b-b99a-2182be14317c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:39.152114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb114315-f29a-4688-aa1a-c431b2a3328a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:39.152709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb114315-f29a-4688-aa1a-c431b2a3328a admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:39.153003 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb114315-f29a-4688-aa1a-c431b2a3328a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:39.153307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb114315-f29a-4688-aa1a-c431b2a3328a admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:39.153938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb114315-f29a-4688-aa1a-c431b2a3328a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:39.155896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb114315-f29a-4688-aa1a-c431b2a3328a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:39.265864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1243/2496] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:57:39 2022] POST /identity/v3/projects => generated 389 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:57:39.270482 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f407fed5-c82d-4cb0-a462-101cc70ed133 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:39.281291 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f407fed5-c82d-4cb0-a462-101cc70ed133 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:39.337369 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26022078-64ed-456a-80b2-48061940f027 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:39.338014 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26022078-64ed-456a-80b2-48061940f027 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:39.338189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26022078-64ed-456a-80b2-48061940f027 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:39.338448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26022078-64ed-456a-80b2-48061940f027 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:39.339076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26022078-64ed-456a-80b2-48061940f027 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:39.341000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26022078-64ed-456a-80b2-48061940f027 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:39.459540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1254/2497] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:57:39 2022] POST /identity/v3/users => generated 301 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:57:39.463612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb114315-f29a-4688-aa1a-c431b2a3328a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:39.474642 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb114315-f29a-4688-aa1a-c431b2a3328a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:39.479467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ec78e1-fe62-423e-b9b6-4dffaff59fc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:39.480052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ec78e1-fe62-423e-b9b6-4dffaff59fc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:39.480326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ec78e1-fe62-423e-b9b6-4dffaff59fc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:39.480592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ec78e1-fe62-423e-b9b6-4dffaff59fc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:39.481222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ec78e1-fe62-423e-b9b6-4dffaff59fc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:39.482825 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ec78e1-fe62-423e-b9b6-4dffaff59fc3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:39.494233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1244/2498] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:39.498400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26022078-64ed-456a-80b2-48061940f027 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:39.508757 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26022078-64ed-456a-80b2-48061940f027 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:39.513250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7261d451-3579-4788-86a1-4e076e2a464f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:39.513884 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7261d451-3579-4788-86a1-4e076e2a464f admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:39.514204 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7261d451-3579-4788-86a1-4e076e2a464f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:39.514501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7261d451-3579-4788-86a1-4e076e2a464f admin admin] PATH_INFO: `/v3/projects/e2012be5a1b84f449a7902189fa58334/users/0a97f7f60e0b4e13bb7b01514f16400c/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:39.529865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7261d451-3579-4788-86a1-4e076e2a464f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2012be5a1b84f449a7902189fa58334, user_id=0a97f7f60e0b4e13bb7b01514f16400c, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:39.533093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7261d451-3579-4788-86a1-4e076e2a464f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:39.608446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1255/2499] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:39 2022] PUT /identity/v3/projects/e2012be5a1b84f449a7902189fa58334/users/0a97f7f60e0b4e13bb7b01514f16400c/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:39.614446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ec78e1-fe62-423e-b9b6-4dffaff59fc3 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:39.623233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ec78e1-fe62-423e-b9b6-4dffaff59fc3 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:39.682235 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de607695-6fec-4c06-8505-6d3a2d58bab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:39.682820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de607695-6fec-4c06-8505-6d3a2d58bab3 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:39.683095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de607695-6fec-4c06-8505-6d3a2d58bab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:39.683361 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de607695-6fec-4c06-8505-6d3a2d58bab3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:39.683991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de607695-6fec-4c06-8505-6d3a2d58bab3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:39.685609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de607695-6fec-4c06-8505-6d3a2d58bab3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:39.696228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1245/2500] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:39 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:39.700572 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7261d451-3579-4788-86a1-4e076e2a464f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:39.711458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7261d451-3579-4788-86a1-4e076e2a464f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:39.716053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe62e264-3361-4505-a0a6-6518d89d85d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:39.716692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe62e264-3361-4505-a0a6-6518d89d85d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:39.716966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe62e264-3361-4505-a0a6-6518d89d85d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:39.717233 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe62e264-3361-4505-a0a6-6518d89d85d2 admin admin] PATH_INFO: `/v3/projects/e2012be5a1b84f449a7902189fa58334/users/0a97f7f60e0b4e13bb7b01514f16400c/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:39.719271 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe62e264-3361-4505-a0a6-6518d89d85d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2012be5a1b84f449a7902189fa58334, user_id=0a97f7f60e0b4e13bb7b01514f16400c, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:39.722317 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe62e264-3361-4505-a0a6-6518d89d85d2 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:39.732961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1256/2501] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:39 2022] PUT /identity/v3/projects/e2012be5a1b84f449a7902189fa58334/users/0a97f7f60e0b4e13bb7b01514f16400c/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:39.745391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c063b943-7009-4dc5-936b-2b0abb49c0ad None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:39.745680 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c063b943-7009-4dc5-936b-2b0abb49c0ad None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:39.745980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c063b943-7009-4dc5-936b-2b0abb49c0ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:39.781030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-c063b943-7009-4dc5-936b-2b0abb49c0ad None None] MFA Rules not processed for user `0a97f7f60e0b4e13bb7b01514f16400c`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:39.782315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c063b943-7009-4dc5-936b-2b0abb49c0ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:39.831795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:39.851784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1246/2502] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:39 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:39.863439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe62e264-3361-4505-a0a6-6518d89d85d2 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:39.865173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-fe62e264-3361-4505-a0a6-6518d89d85d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:39.914057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:39.925659 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe62e264-3361-4505-a0a6-6518d89d85d2 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:39.929625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92c499c-2421-4901-9f1f-1b85a3a9efd5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:39.930283 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92c499c-2421-4901-9f1f-1b85a3a9efd5 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:39.930560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92c499c-2421-4901-9f1f-1b85a3a9efd5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:39.930827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92c499c-2421-4901-9f1f-1b85a3a9efd5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:39.940770 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92c499c-2421-4901-9f1f-1b85a3a9efd5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:39.942229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92c499c-2421-4901-9f1f-1b85a3a9efd5 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:39.951516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1257/2503] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:39 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:40.089985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c063b943-7009-4dc5-936b-2b0abb49c0ad None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:40.152603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c063b943-7009-4dc5-936b-2b0abb49c0ad None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:40.157231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:40.158451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:40.158728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:40.159000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] PATH_INFO: `/v3/users/18985578c86f442997471fea0df42e15` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:40.160228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=18985578c86f442997471fea0df42e15)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:40.162217 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:40.375437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18985578c86f442997471fea0df42e15'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:40.376140 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18985578c86f442997471fea0df42e15'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:40.383566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18985578c86f442997471fea0df42e15'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:40.396028 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18985578c86f442997471fea0df42e15'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:40.498479 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18985578c86f442997471fea0df42e15'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:40.505270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1247/2504] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:57:40 2022] DELETE /identity/v3/users/18985578c86f442997471fea0df42e15 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:40.509796 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588d55c7-93a9-4120-ba7d-0bb991ad7ef5 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:40.510278 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588d55c7-93a9-4120-ba7d-0bb991ad7ef5 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:40.510421 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588d55c7-93a9-4120-ba7d-0bb991ad7ef5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:40.530411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-588d55c7-93a9-4120-ba7d-0bb991ad7ef5 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:40.531618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-588d55c7-93a9-4120-ba7d-0bb991ad7ef5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:40.591270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1258/2505] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:40.601707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:40.603399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:40.664164 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178bb0d2-e863-4591-9aec-2d28e2395ea6 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:40.667863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e877ae-bd44-492b-be7c-f7564783b134 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:40.668405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e877ae-bd44-492b-be7c-f7564783b134 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:40.668714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e877ae-bd44-492b-be7c-f7564783b134 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:40.668979 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e877ae-bd44-492b-be7c-f7564783b134 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:40.678272 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e877ae-bd44-492b-be7c-f7564783b134 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:40.679699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e877ae-bd44-492b-be7c-f7564783b134 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:40.689639 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1248/2506] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:40.795478 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588d55c7-93a9-4120-ba7d-0bb991ad7ef5 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:40.796951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-588d55c7-93a9-4120-ba7d-0bb991ad7ef5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:40.855416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588d55c7-93a9-4120-ba7d-0bb991ad7ef5 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:40.859117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8507967e-22f0-4887-8511-ee4649b328a4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:40.859726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8507967e-22f0-4887-8511-ee4649b328a4 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:40.859999 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8507967e-22f0-4887-8511-ee4649b328a4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:40.860262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8507967e-22f0-4887-8511-ee4649b328a4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:40.862939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8507967e-22f0-4887-8511-ee4649b328a4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:40.933657 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8507967e-22f0-4887-8511-ee4649b328a4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:40.935059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8507967e-22f0-4887-8511-ee4649b328a4 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:40.945782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1259/2507] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:40 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:41.118537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e877ae-bd44-492b-be7c-f7564783b134 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:41.121750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-45e877ae-bd44-492b-be7c-f7564783b134 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:41.136794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e877ae-bd44-492b-be7c-f7564783b134 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:41.143228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:41.143565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:41.143849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:41.144120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] PATH_INFO: `/v3/projects/6bdd915634de4482b0f7022dd0d3e0de` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:41.145769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] RBAC: Authorizing `identity:delete_project(project_id=6bdd915634de4482b0f7022dd0d3e0de)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:41.147898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:41.150875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:41.266090 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6bdd915634de4482b0f7022dd0d3e0de', 'request_id': 'req-1de65794-4729-4581-8fae-7cb667e35e1a'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:41.272306 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6bdd915634de4482b0f7022dd0d3e0de', 'request_id': 'req-1de65794-4729-4581-8fae-7cb667e35e1a'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:41.278826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6bdd915634de4482b0f7022dd0d3e0de', 'request_id': 'req-1de65794-4729-4581-8fae-7cb667e35e1a'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:41.401136 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] The token cache is being invalidate because project 6bdd915634de4482b0f7022dd0d3e0de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:57:41.401497 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:41.402124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:41.403724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1249/2508] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:57:41 2022] DELETE /identity/v3/projects/6bdd915634de4482b0f7022dd0d3e0de => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:41.408797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8507967e-22f0-4887-8511-ee4649b328a4 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:41.420290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8507967e-22f0-4887-8511-ee4649b328a4 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:41.453899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:41.477768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef6eccc-85eb-411d-a747-4d3d75b27cce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:41.478680 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef6eccc-85eb-411d-a747-4d3d75b27cce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:41.478961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef6eccc-85eb-411d-a747-4d3d75b27cce admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:41.479231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef6eccc-85eb-411d-a747-4d3d75b27cce admin admin] PATH_INFO: `/v3/users/0a97f7f60e0b4e13bb7b01514f16400c` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:41.480357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef6eccc-85eb-411d-a747-4d3d75b27cce admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a97f7f60e0b4e13bb7b01514f16400c)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:41.482420 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef6eccc-85eb-411d-a747-4d3d75b27cce admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:41.706951 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-7ef6eccc-85eb-411d-a747-4d3d75b27cce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a97f7f60e0b4e13bb7b01514f16400c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:41.707606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-7ef6eccc-85eb-411d-a747-4d3d75b27cce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a97f7f60e0b4e13bb7b01514f16400c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:41.714511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-7ef6eccc-85eb-411d-a747-4d3d75b27cce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a97f7f60e0b4e13bb7b01514f16400c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:41.715139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-7ef6eccc-85eb-411d-a747-4d3d75b27cce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a97f7f60e0b4e13bb7b01514f16400c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:41.801423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-7ef6eccc-85eb-411d-a747-4d3d75b27cce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a97f7f60e0b4e13bb7b01514f16400c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:41.820299 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1260/2509] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:57:41 2022] DELETE /identity/v3/users/0a97f7f60e0b4e13bb7b01514f16400c => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:42.294630 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:42.298877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:42.309433 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de65794-4729-4581-8fae-7cb667e35e1a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:42.368466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ef2d69-698f-4cac-8b9a-f4e24acda363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:42.369101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ef2d69-698f-4cac-8b9a-f4e24acda363 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:42.369417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ef2d69-698f-4cac-8b9a-f4e24acda363 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:42.369689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ef2d69-698f-4cac-8b9a-f4e24acda363 admin admin] PATH_INFO: `/v3/projects/e2012be5a1b84f449a7902189fa58334` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:42.370827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ef2d69-698f-4cac-8b9a-f4e24acda363 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2012be5a1b84f449a7902189fa58334)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:42.372811 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ef2d69-698f-4cac-8b9a-f4e24acda363 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:42.493592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-72ef2d69-698f-4cac-8b9a-f4e24acda363 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2012be5a1b84f449a7902189fa58334', 'request_id': 'req-72ef2d69-698f-4cac-8b9a-f4e24acda363'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:42.499107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-72ef2d69-698f-4cac-8b9a-f4e24acda363 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2012be5a1b84f449a7902189fa58334', 'request_id': 'req-72ef2d69-698f-4cac-8b9a-f4e24acda363'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:42.506372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-72ef2d69-698f-4cac-8b9a-f4e24acda363 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2012be5a1b84f449a7902189fa58334', 'request_id': 'req-72ef2d69-698f-4cac-8b9a-f4e24acda363'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:42.613183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-72ef2d69-698f-4cac-8b9a-f4e24acda363 admin admin] The token cache is being invalidate because project e2012be5a1b84f449a7902189fa58334 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:57:42.613471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-72ef2d69-698f-4cac-8b9a-f4e24acda363 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:42.614095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-72ef2d69-698f-4cac-8b9a-f4e24acda363 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:42.615656 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1250/2510] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:57:42 2022] DELETE /identity/v3/projects/e2012be5a1b84f449a7902189fa58334 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:42.620697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58651642-010b-4b14-a5dc-c168d68baa3b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:42.620989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58651642-010b-4b14-a5dc-c168d68baa3b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:42.621268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58651642-010b-4b14-a5dc-c168d68baa3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:42.641736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-58651642-010b-4b14-a5dc-c168d68baa3b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:42.643013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-58651642-010b-4b14-a5dc-c168d68baa3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:42.706743 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1261/2511] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:42.714919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d20c83-5f8e-4f2a-8145-ad3809653b15 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:42.715202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d20c83-5f8e-4f2a-8145-ad3809653b15 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:42.715474 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d20c83-5f8e-4f2a-8145-ad3809653b15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:42.736589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-49d20c83-5f8e-4f2a-8145-ad3809653b15 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:42.737846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-49d20c83-5f8e-4f2a-8145-ad3809653b15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:42.747360 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1251/2512] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:42.752592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58651642-010b-4b14-a5dc-c168d68baa3b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:42.764549 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58651642-010b-4b14-a5dc-c168d68baa3b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:42.769788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff84e23-f538-4491-be16-64ab0f8ac400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:42.770434 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff84e23-f538-4491-be16-64ab0f8ac400 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:42.770754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff84e23-f538-4491-be16-64ab0f8ac400 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:42.771023 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff84e23-f538-4491-be16-64ab0f8ac400 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:42.771682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff84e23-f538-4491-be16-64ab0f8ac400 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:57:42.771983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff84e23-f538-4491-be16-64ab0f8ac400 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:42.773590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff84e23-f538-4491-be16-64ab0f8ac400 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:42.786639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1262/2513] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:57:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:57:42.791655 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d20c83-5f8e-4f2a-8145-ad3809653b15 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:42.801331 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d20c83-5f8e-4f2a-8145-ad3809653b15 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:42.806088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040bcb0c-520b-4d23-b615-02a10fc1bf09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:42.806708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040bcb0c-520b-4d23-b615-02a10fc1bf09 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:42.806985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040bcb0c-520b-4d23-b615-02a10fc1bf09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:42.807254 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040bcb0c-520b-4d23-b615-02a10fc1bf09 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:42.807880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040bcb0c-520b-4d23-b615-02a10fc1bf09 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:42.809761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040bcb0c-520b-4d23-b615-02a10fc1bf09 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:42.824919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:42.905427 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1252/2514] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:57:42 2022] POST /identity/v3/projects => generated 377 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:57:42.910229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff84e23-f538-4491-be16-64ab0f8ac400 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:42.920473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff84e23-f538-4491-be16-64ab0f8ac400 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:42.967786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:42.982719 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3bb436-c3ff-45e8-8960-7296ffd5fdb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:42.983373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3bb436-c3ff-45e8-8960-7296ffd5fdb5 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:42.983655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3bb436-c3ff-45e8-8960-7296ffd5fdb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:42.983938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3bb436-c3ff-45e8-8960-7296ffd5fdb5 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:42.984574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3bb436-c3ff-45e8-8960-7296ffd5fdb5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:42.986903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3bb436-c3ff-45e8-8960-7296ffd5fdb5 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:43.071319 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1263/2515] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:57:42 2022] POST /identity/v3/users => generated 289 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:57:43.075536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040bcb0c-520b-4d23-b615-02a10fc1bf09 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:43.089401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040bcb0c-520b-4d23-b615-02a10fc1bf09 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:43.095604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592c0b12-3c5d-46df-a615-860f3957c625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:43.096549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592c0b12-3c5d-46df-a615-860f3957c625 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:43.097013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592c0b12-3c5d-46df-a615-860f3957c625 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:43.097467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592c0b12-3c5d-46df-a615-860f3957c625 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:43.098582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592c0b12-3c5d-46df-a615-860f3957c625 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:43.101342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592c0b12-3c5d-46df-a615-860f3957c625 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:43.113197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1253/2516] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:43 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:43.117462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3bb436-c3ff-45e8-8960-7296ffd5fdb5 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:43.129363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3bb436-c3ff-45e8-8960-7296ffd5fdb5 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:43.134226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6b20ec-0943-4a68-b80a-d886747e13db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:43.134915 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6b20ec-0943-4a68-b80a-d886747e13db admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:43.135195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6b20ec-0943-4a68-b80a-d886747e13db admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:43.135463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6b20ec-0943-4a68-b80a-d886747e13db admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:43.136437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6b20ec-0943-4a68-b80a-d886747e13db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:43.138190 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6b20ec-0943-4a68-b80a-d886747e13db admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:43.151456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1264/2517] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:43 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:43.155856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592c0b12-3c5d-46df-a615-860f3957c625 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:43.166851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592c0b12-3c5d-46df-a615-860f3957c625 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:43.171778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7ef6f7-46aa-4bd7-9ad7-24e99ba51bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:43.172438 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7ef6f7-46aa-4bd7-9ad7-24e99ba51bfc admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:43.172718 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7ef6f7-46aa-4bd7-9ad7-24e99ba51bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:43.177050 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7ef6f7-46aa-4bd7-9ad7-24e99ba51bfc admin admin] PATH_INFO: `/v3/projects/965e207f27d3407ba2e20a8c2e6fed02/users/93c48919fdfb4a75a64f75f23305fa70/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:43.192413 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7ef6f7-46aa-4bd7-9ad7-24e99ba51bfc admin admin] RBAC: Authorizing `identity:create_grant(project_id=965e207f27d3407ba2e20a8c2e6fed02, user_id=93c48919fdfb4a75a64f75f23305fa70, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:43.195236 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7ef6f7-46aa-4bd7-9ad7-24e99ba51bfc admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:43.259623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1254/2518] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:43 2022] PUT /identity/v3/projects/965e207f27d3407ba2e20a8c2e6fed02/users/93c48919fdfb4a75a64f75f23305fa70/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:43.270985 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76135b3d-429b-473e-b8e5-2c432f0743a0 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:43.271387 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76135b3d-429b-473e-b8e5-2c432f0743a0 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:43.271667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76135b3d-429b-473e-b8e5-2c432f0743a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:43.301428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-76135b3d-429b-473e-b8e5-2c432f0743a0 None None] MFA Rules not processed for user `93c48919fdfb4a75a64f75f23305fa70`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:43.302753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-76135b3d-429b-473e-b8e5-2c432f0743a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:43.376315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1265/2519] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:43 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:43.381659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7ef6f7-46aa-4bd7-9ad7-24e99ba51bfc admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:43.459315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7ef6f7-46aa-4bd7-9ad7-24e99ba51bfc admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:43.464475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c46e1f6-7633-4726-931e-5c2d6d403ef8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:43.465079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c46e1f6-7633-4726-931e-5c2d6d403ef8 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:43.465365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c46e1f6-7633-4726-931e-5c2d6d403ef8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:43.465638 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c46e1f6-7633-4726-931e-5c2d6d403ef8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:43.466302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c46e1f6-7633-4726-931e-5c2d6d403ef8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:43.468216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c46e1f6-7633-4726-931e-5c2d6d403ef8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:43.562104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1255/2520] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:57:43 2022] POST /identity/v3/projects => generated 379 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:57:43.567375 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76135b3d-429b-473e-b8e5-2c432f0743a0 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:43.578731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76135b3d-429b-473e-b8e5-2c432f0743a0 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:43.640457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56e44f4-ebdf-4e85-86d6-0f85d3900b16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:43.640986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56e44f4-ebdf-4e85-86d6-0f85d3900b16 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:43.641253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56e44f4-ebdf-4e85-86d6-0f85d3900b16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:43.641528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56e44f4-ebdf-4e85-86d6-0f85d3900b16 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:43.642169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56e44f4-ebdf-4e85-86d6-0f85d3900b16 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:43.644188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56e44f4-ebdf-4e85-86d6-0f85d3900b16 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:43.750434 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1266/2521] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:57:43 2022] POST /identity/v3/users => generated 290 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:57:43.754362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c46e1f6-7633-4726-931e-5c2d6d403ef8 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:43.764751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c46e1f6-7633-4726-931e-5c2d6d403ef8 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:43.769589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33237f-9767-4e69-95c7-1f9144eadea4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:43.770217 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33237f-9767-4e69-95c7-1f9144eadea4 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:43.770501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33237f-9767-4e69-95c7-1f9144eadea4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:43.770774 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33237f-9767-4e69-95c7-1f9144eadea4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:43.771495 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa33237f-9767-4e69-95c7-1f9144eadea4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:43.773189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa33237f-9767-4e69-95c7-1f9144eadea4 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:43.784753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1256/2522] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:43.788699 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56e44f4-ebdf-4e85-86d6-0f85d3900b16 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:43.800280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56e44f4-ebdf-4e85-86d6-0f85d3900b16 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:43.804939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f028c0-2459-4ed2-a8b4-b32411521d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:43.805538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f028c0-2459-4ed2-a8b4-b32411521d06 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:43.805811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f028c0-2459-4ed2-a8b4-b32411521d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:43.806077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f028c0-2459-4ed2-a8b4-b32411521d06 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:43.806747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f028c0-2459-4ed2-a8b4-b32411521d06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:43.808326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f028c0-2459-4ed2-a8b4-b32411521d06 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:43.821695 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1267/2523] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:43.823353 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33237f-9767-4e69-95c7-1f9144eadea4 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:43.833892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33237f-9767-4e69-95c7-1f9144eadea4 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:43.838641 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e54d51-1cb0-4557-8a66-b45b5e403c19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:43.839288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e54d51-1cb0-4557-8a66-b45b5e403c19 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:43.839566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e54d51-1cb0-4557-8a66-b45b5e403c19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:43.839839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e54d51-1cb0-4557-8a66-b45b5e403c19 admin admin] PATH_INFO: `/v3/projects/aadee20092144a1e806deefa3e05e85f/users/70bbe04d52634c4cb16cfa0896eea8f6/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:43.855026 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e54d51-1cb0-4557-8a66-b45b5e403c19 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aadee20092144a1e806deefa3e05e85f, user_id=70bbe04d52634c4cb16cfa0896eea8f6, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:43.857717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e54d51-1cb0-4557-8a66-b45b5e403c19 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:43.929157 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1257/2524] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:43 2022] PUT /identity/v3/projects/aadee20092144a1e806deefa3e05e85f/users/70bbe04d52634c4cb16cfa0896eea8f6/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:57:43.940850 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1803b86d-6d93-4244-9dc7-ef73b8889694 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:43.941134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1803b86d-6d93-4244-9dc7-ef73b8889694 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:43.941408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1803b86d-6d93-4244-9dc7-ef73b8889694 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:43.969136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1803b86d-6d93-4244-9dc7-ef73b8889694 None None] MFA Rules not processed for user `70bbe04d52634c4cb16cfa0896eea8f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:43.971000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1803b86d-6d93-4244-9dc7-ef73b8889694 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:43.990384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:44.035787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1268/2525] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:43 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:57:44.040473 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e54d51-1cb0-4557-8a66-b45b5e403c19 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:44.102396 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e54d51-1cb0-4557-8a66-b45b5e403c19 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:44.107418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0470b1ef-c1c9-4d0d-8a09-0c00e39d50c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:44.108014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0470b1ef-c1c9-4d0d-8a09-0c00e39d50c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:44.108296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0470b1ef-c1c9-4d0d-8a09-0c00e39d50c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:44.108571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0470b1ef-c1c9-4d0d-8a09-0c00e39d50c9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:44.109199 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0470b1ef-c1c9-4d0d-8a09-0c00e39d50c9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:44.111380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0470b1ef-c1c9-4d0d-8a09-0c00e39d50c9 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:44.125201 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:44.190966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1258/2526] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:57:44 2022] POST /identity/v3/projects => generated 377 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:57:44.195000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1803b86d-6d93-4244-9dc7-ef73b8889694 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:44.206499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1803b86d-6d93-4244-9dc7-ef73b8889694 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:44.261961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bb4d69-56db-44c7-84cd-6d4660457ad3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:44.262590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bb4d69-56db-44c7-84cd-6d4660457ad3 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:44.262866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bb4d69-56db-44c7-84cd-6d4660457ad3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:44.263133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bb4d69-56db-44c7-84cd-6d4660457ad3 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:44.263744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bb4d69-56db-44c7-84cd-6d4660457ad3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:44.265647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bb4d69-56db-44c7-84cd-6d4660457ad3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:44.343713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1269/2527] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:57:44 2022] POST /identity/v3/users => generated 295 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:57:44.348596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0470b1ef-c1c9-4d0d-8a09-0c00e39d50c9 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:44.359471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0470b1ef-c1c9-4d0d-8a09-0c00e39d50c9 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:44.364294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0e03e6-e480-41a3-b22e-fe4e3f70f37b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:44.364891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0e03e6-e480-41a3-b22e-fe4e3f70f37b admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:44.365173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0e03e6-e480-41a3-b22e-fe4e3f70f37b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:44.365447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0e03e6-e480-41a3-b22e-fe4e3f70f37b admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:44.366116 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0e03e6-e480-41a3-b22e-fe4e3f70f37b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:44.367731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0e03e6-e480-41a3-b22e-fe4e3f70f37b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:44.378658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1259/2528] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:44.385740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bb4d69-56db-44c7-84cd-6d4660457ad3 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:44.396797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bb4d69-56db-44c7-84cd-6d4660457ad3 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:44.401597 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a96ac0-26a1-40c4-99c5-961df1fe0f2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:44.402278 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a96ac0-26a1-40c4-99c5-961df1fe0f2a admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:44.402561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a96ac0-26a1-40c4-99c5-961df1fe0f2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:44.402831 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a96ac0-26a1-40c4-99c5-961df1fe0f2a admin admin] PATH_INFO: `/v3/projects/1d387c929a0c40eb9b76f98dc3f877fe/users/21864706084b46bf88c4c04fa9cd1b6c/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:44.418384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a96ac0-26a1-40c4-99c5-961df1fe0f2a admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d387c929a0c40eb9b76f98dc3f877fe, user_id=21864706084b46bf88c4c04fa9cd1b6c, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:44.421036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a96ac0-26a1-40c4-99c5-961df1fe0f2a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:44.487236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1270/2529] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:44 2022] PUT /identity/v3/projects/1d387c929a0c40eb9b76f98dc3f877fe/users/21864706084b46bf88c4c04fa9cd1b6c/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:44.491225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0e03e6-e480-41a3-b22e-fe4e3f70f37b admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:44.501619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0e03e6-e480-41a3-b22e-fe4e3f70f37b admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:44.558894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bdee63-5fc9-4fdf-9e04-63916e7a77d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:44.559506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bdee63-5fc9-4fdf-9e04-63916e7a77d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:44.559851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bdee63-5fc9-4fdf-9e04-63916e7a77d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:44.560162 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bdee63-5fc9-4fdf-9e04-63916e7a77d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:44.560803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bdee63-5fc9-4fdf-9e04-63916e7a77d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:44.562437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bdee63-5fc9-4fdf-9e04-63916e7a77d1 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:44.572847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1260/2530] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:44 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:57:44.577029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a96ac0-26a1-40c4-99c5-961df1fe0f2a admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:44.588106 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a96ac0-26a1-40c4-99c5-961df1fe0f2a admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:44.592912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26ef6e1-99e8-4044-ab5e-1eaf0e6c07ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:44.593560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26ef6e1-99e8-4044-ab5e-1eaf0e6c07ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:44.593842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26ef6e1-99e8-4044-ab5e-1eaf0e6c07ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:44.594117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26ef6e1-99e8-4044-ab5e-1eaf0e6c07ae admin admin] PATH_INFO: `/v3/projects/1d387c929a0c40eb9b76f98dc3f877fe/users/21864706084b46bf88c4c04fa9cd1b6c/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:44.596965 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26ef6e1-99e8-4044-ab5e-1eaf0e6c07ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d387c929a0c40eb9b76f98dc3f877fe, user_id=21864706084b46bf88c4c04fa9cd1b6c, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:44.599667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26ef6e1-99e8-4044-ab5e-1eaf0e6c07ae admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:44.610299 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1271/2531] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:44 2022] PUT /identity/v3/projects/1d387c929a0c40eb9b76f98dc3f877fe/users/21864706084b46bf88c4c04fa9cd1b6c/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:44.621196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa02afb-fc40-4d73-a669-4cecaf6dbe0b None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:44.621485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa02afb-fc40-4d73-a669-4cecaf6dbe0b None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:44.621765 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa02afb-fc40-4d73-a669-4cecaf6dbe0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:44.649312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-dfa02afb-fc40-4d73-a669-4cecaf6dbe0b None None] MFA Rules not processed for user `21864706084b46bf88c4c04fa9cd1b6c`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:44.650569 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-dfa02afb-fc40-4d73-a669-4cecaf6dbe0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:44.717437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1261/2532] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:44 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:44.729335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26ef6e1-99e8-4044-ab5e-1eaf0e6c07ae admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:44.731057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d26ef6e1-99e8-4044-ab5e-1eaf0e6c07ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:44.790572 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26ef6e1-99e8-4044-ab5e-1eaf0e6c07ae admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:44.794375 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc0cc4e-fea6-46ec-bebe-64b93e094045 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:44.795155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc0cc4e-fea6-46ec-bebe-64b93e094045 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:44.795438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc0cc4e-fea6-46ec-bebe-64b93e094045 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:44.795707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc0cc4e-fea6-46ec-bebe-64b93e094045 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:44.806237 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc0cc4e-fea6-46ec-bebe-64b93e094045 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:44.807679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc0cc4e-fea6-46ec-bebe-64b93e094045 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:44.817373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1272/2533] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:44 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:44.908978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa02afb-fc40-4d73-a669-4cecaf6dbe0b None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:44.910539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-dfa02afb-fc40-4d73-a669-4cecaf6dbe0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:44.924034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa02afb-fc40-4d73-a669-4cecaf6dbe0b None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:44.927907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32c4007-198d-4335-a6bc-39c6b7d8cd27 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:44.928500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32c4007-198d-4335-a6bc-39c6b7d8cd27 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:44.928792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32c4007-198d-4335-a6bc-39c6b7d8cd27 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:44.929044 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32c4007-198d-4335-a6bc-39c6b7d8cd27 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:44.992215 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32c4007-198d-4335-a6bc-39c6b7d8cd27 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:44.993674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32c4007-198d-4335-a6bc-39c6b7d8cd27 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:45.003179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1262/2534] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:44 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:52.187544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc0cc4e-fea6-46ec-bebe-64b93e094045 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:52.205428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:52.252210 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc0cc4e-fea6-46ec-bebe-64b93e094045 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:52.257310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:52.257876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:52.258178 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:52.258440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] PATH_INFO: `/v3/users/93c48919fdfb4a75a64f75f23305fa70` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:52.259583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] RBAC: Authorizing `identity:delete_user(user_id=93c48919fdfb4a75a64f75f23305fa70)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:52.261514 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:52.484057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '93c48919fdfb4a75a64f75f23305fa70'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:52.484720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '93c48919fdfb4a75a64f75f23305fa70'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:52.491560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '93c48919fdfb4a75a64f75f23305fa70'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:52.492257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '93c48919fdfb4a75a64f75f23305fa70'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:52.578742 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '93c48919fdfb4a75a64f75f23305fa70'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:52.596847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1273/2535] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:57:52 2022] DELETE /identity/v3/users/93c48919fdfb4a75a64f75f23305fa70 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:52.602167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8e9ea1-e3b6-4bfe-8c5e-df1e53d5636b None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:52.602635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8e9ea1-e3b6-4bfe-8c5e-df1e53d5636b None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:52.602778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8e9ea1-e3b6-4bfe-8c5e-df1e53d5636b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:52.610108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:52.623600 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-cf8e9ea1-e3b6-4bfe-8c5e-df1e53d5636b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:52.624820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-cf8e9ea1-e3b6-4bfe-8c5e-df1e53d5636b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:52.683999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1263/2536] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:52.694544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:52.695948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:52.755365 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2a1b0a-86cf-420c-b68a-c96b148ed7ff admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:52.759414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470e2d21-1bb2-4b90-947f-fa47ceda052c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:52.760019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470e2d21-1bb2-4b90-947f-fa47ceda052c service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:52.760292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470e2d21-1bb2-4b90-947f-fa47ceda052c service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:52.760560 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470e2d21-1bb2-4b90-947f-fa47ceda052c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:52.769327 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470e2d21-1bb2-4b90-947f-fa47ceda052c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:52.771855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470e2d21-1bb2-4b90-947f-fa47ceda052c service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:52.781275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1274/2537] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:53.226424 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8e9ea1-e3b6-4bfe-8c5e-df1e53d5636b None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:53.227913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-cf8e9ea1-e3b6-4bfe-8c5e-df1e53d5636b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:53.241265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8e9ea1-e3b6-4bfe-8c5e-df1e53d5636b None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:53.246092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:53.246750 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:53.247035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:53.247305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] PATH_INFO: `/v3/projects/965e207f27d3407ba2e20a8c2e6fed02` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:53.248437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=965e207f27d3407ba2e20a8c2e6fed02)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:53.250922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:53.379773 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '965e207f27d3407ba2e20a8c2e6fed02', 'request_id': 'req-eea25bd5-4af1-4fb2-9383-12d496a828fa'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:53.385376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '965e207f27d3407ba2e20a8c2e6fed02', 'request_id': 'req-eea25bd5-4af1-4fb2-9383-12d496a828fa'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:53.391586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '965e207f27d3407ba2e20a8c2e6fed02', 'request_id': 'req-eea25bd5-4af1-4fb2-9383-12d496a828fa'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:53.549751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] The token cache is being invalidate because project 965e207f27d3407ba2e20a8c2e6fed02 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:57:53.550044 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:53.555775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:53.555775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1264/2538] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:57:53 2022] DELETE /identity/v3/projects/965e207f27d3407ba2e20a8c2e6fed02 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:53.557240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470e2d21-1bb2-4b90-947f-fa47ceda052c service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:53.566976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470e2d21-1bb2-4b90-947f-fa47ceda052c service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:53.600813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:53.625771 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:53.626438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:53.626721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:53.626997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] PATH_INFO: `/v3/users/70bbe04d52634c4cb16cfa0896eea8f6` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:53.628182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] RBAC: Authorizing `identity:delete_user(user_id=70bbe04d52634c4cb16cfa0896eea8f6)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:53.630459 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:53.872159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '70bbe04d52634c4cb16cfa0896eea8f6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:53.872899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '70bbe04d52634c4cb16cfa0896eea8f6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:53.879695 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '70bbe04d52634c4cb16cfa0896eea8f6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:53.880540 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '70bbe04d52634c4cb16cfa0896eea8f6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:53.977312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '70bbe04d52634c4cb16cfa0896eea8f6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:53.996674 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1275/2539] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:57:53 2022] DELETE /identity/v3/users/70bbe04d52634c4cb16cfa0896eea8f6 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:54.403233 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:54.404766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:54.419550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea25bd5-4af1-4fb2-9383-12d496a828fa admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:54.479291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:54.479916 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:54.480195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:54.480466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] PATH_INFO: `/v3/projects/aadee20092144a1e806deefa3e05e85f` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:54.481668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aadee20092144a1e806deefa3e05e85f)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:54.483613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:54.570062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:54.585075 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aadee20092144a1e806deefa3e05e85f', 'request_id': 'req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:54.590733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aadee20092144a1e806deefa3e05e85f', 'request_id': 'req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:54.596863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aadee20092144a1e806deefa3e05e85f', 'request_id': 'req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:54.714123 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] The token cache is being invalidate because project aadee20092144a1e806deefa3e05e85f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:57:54.714382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:54.715012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:54.716538 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1265/2540] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:57:54 2022] DELETE /identity/v3/projects/aadee20092144a1e806deefa3e05e85f => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:54.720665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:54.723823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:54.735043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ceaf29-689f-4e0b-8af3-08e4f3fad335 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:54.800401 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c193ea-de16-4aed-91c1-7e2c56ac5204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:54.801088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c193ea-de16-4aed-91c1-7e2c56ac5204 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:54.801372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c193ea-de16-4aed-91c1-7e2c56ac5204 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:54.802197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c193ea-de16-4aed-91c1-7e2c56ac5204 admin admin] PATH_INFO: `/v3/users/21864706084b46bf88c4c04fa9cd1b6c` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:54.803738 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c193ea-de16-4aed-91c1-7e2c56ac5204 admin admin] RBAC: Authorizing `identity:delete_user(user_id=21864706084b46bf88c4c04fa9cd1b6c)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:54.805690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c193ea-de16-4aed-91c1-7e2c56ac5204 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:54.980245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-27c193ea-de16-4aed-91c1-7e2c56ac5204 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21864706084b46bf88c4c04fa9cd1b6c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:54.980833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-27c193ea-de16-4aed-91c1-7e2c56ac5204 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21864706084b46bf88c4c04fa9cd1b6c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:54.987789 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-27c193ea-de16-4aed-91c1-7e2c56ac5204 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21864706084b46bf88c4c04fa9cd1b6c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:54.988417 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-27c193ea-de16-4aed-91c1-7e2c56ac5204 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21864706084b46bf88c4c04fa9cd1b6c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:54.991883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:55.087077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-27c193ea-de16-4aed-91c1-7e2c56ac5204 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21864706084b46bf88c4c04fa9cd1b6c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:55.105670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1276/2541] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:57:54 2022] DELETE /identity/v3/users/21864706084b46bf88c4c04fa9cd1b6c => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:55.539929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:55.546255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:55.556031 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b30ac4-0fb7-4fd0-a53e-8feba9ee29f3 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:55.615946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:55.616575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:55.616855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:55.617172 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f admin admin] PATH_INFO: `/v3/projects/1d387c929a0c40eb9b76f98dc3f877fe` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:55.618336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d387c929a0c40eb9b76f98dc3f877fe)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:55.621642 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:55.715373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d387c929a0c40eb9b76f98dc3f877fe', 'request_id': 'req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:55.720879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d387c929a0c40eb9b76f98dc3f877fe', 'request_id': 'req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:55.727817 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d387c929a0c40eb9b76f98dc3f877fe', 'request_id': 'req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:55.812245 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f admin admin] The token cache is being invalidate because project 1d387c929a0c40eb9b76f98dc3f877fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:57:55.812548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:55.813205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-85b79b82-6ad9-4dd1-9fb3-90049a72dc0f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:55.814732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1266/2542] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:57:55 2022] DELETE /identity/v3/projects/1d387c929a0c40eb9b76f98dc3f877fe => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:55.820759 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddace393-7509-4ec8-a16a-effb50fe3a8d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:55.821534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddace393-7509-4ec8-a16a-effb50fe3a8d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:55.821534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddace393-7509-4ec8-a16a-effb50fe3a8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:55.842735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-ddace393-7509-4ec8-a16a-effb50fe3a8d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:55.843954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ddace393-7509-4ec8-a16a-effb50fe3a8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:55.903441 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1277/2543] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:55.912499 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bc6fdf-b3eb-4228-9a9d-09fea931f3e9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:55.912813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bc6fdf-b3eb-4228-9a9d-09fea931f3e9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:55.913114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bc6fdf-b3eb-4228-9a9d-09fea931f3e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:55.934539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-52bc6fdf-b3eb-4228-9a9d-09fea931f3e9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:55.935761 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-52bc6fdf-b3eb-4228-9a9d-09fea931f3e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:55.943740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1267/2544] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:55.949244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddace393-7509-4ec8-a16a-effb50fe3a8d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:55.959404 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddace393-7509-4ec8-a16a-effb50fe3a8d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:55.963970 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f994dfd4-4110-472e-8486-4172af6c41ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:55.964565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f994dfd4-4110-472e-8486-4172af6c41ad admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:55.964841 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f994dfd4-4110-472e-8486-4172af6c41ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:55.965111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f994dfd4-4110-472e-8486-4172af6c41ad admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:55.965748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f994dfd4-4110-472e-8486-4172af6c41ad admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:57:55.966047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f994dfd4-4110-472e-8486-4172af6c41ad admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:55.967687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f994dfd4-4110-472e-8486-4172af6c41ad admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:55.979903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1278/2545] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:57:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:57:55.984443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bc6fdf-b3eb-4228-9a9d-09fea931f3e9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:55.993646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bc6fdf-b3eb-4228-9a9d-09fea931f3e9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:55.998431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c951af94-c560-40eb-8503-0d7ebbf8ebe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:55.999144 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c951af94-c560-40eb-8503-0d7ebbf8ebe0 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:55.999428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c951af94-c560-40eb-8503-0d7ebbf8ebe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:55.999699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c951af94-c560-40eb-8503-0d7ebbf8ebe0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:56.000327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c951af94-c560-40eb-8503-0d7ebbf8ebe0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:56.002246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c951af94-c560-40eb-8503-0d7ebbf8ebe0 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:56.088849 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1268/2546] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:57:55 2022] POST /identity/v3/projects => generated 373 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:57:56.093000 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f994dfd4-4110-472e-8486-4172af6c41ad admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:56.104187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f994dfd4-4110-472e-8486-4172af6c41ad admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:56.159174 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add1a72f-d3cd-4eee-b413-76a33393c077 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:56.159781 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add1a72f-d3cd-4eee-b413-76a33393c077 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:56.160064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add1a72f-d3cd-4eee-b413-76a33393c077 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:56.160336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add1a72f-d3cd-4eee-b413-76a33393c077 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:56.161076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add1a72f-d3cd-4eee-b413-76a33393c077 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:56.163571 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add1a72f-d3cd-4eee-b413-76a33393c077 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:56.252918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1279/2547] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:57:56 2022] POST /identity/v3/users => generated 287 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:57:56.257196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c951af94-c560-40eb-8503-0d7ebbf8ebe0 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:56.267784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c951af94-c560-40eb-8503-0d7ebbf8ebe0 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:56.272986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bb192a-67d6-4208-94a1-6f4246f86bc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:56.273577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb192a-67d6-4208-94a1-6f4246f86bc7 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:56.273939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb192a-67d6-4208-94a1-6f4246f86bc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:56.274274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb192a-67d6-4208-94a1-6f4246f86bc7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:56.274922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bb192a-67d6-4208-94a1-6f4246f86bc7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:56.276506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bb192a-67d6-4208-94a1-6f4246f86bc7 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:56.287342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1269/2548] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:56.291250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add1a72f-d3cd-4eee-b413-76a33393c077 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:56.301380 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add1a72f-d3cd-4eee-b413-76a33393c077 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:56.306911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8929a2d3-86a8-498b-ac6f-6b844a5e4acf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:56.307504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8929a2d3-86a8-498b-ac6f-6b844a5e4acf admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:56.307787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8929a2d3-86a8-498b-ac6f-6b844a5e4acf admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:56.308064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8929a2d3-86a8-498b-ac6f-6b844a5e4acf admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:56.308707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8929a2d3-86a8-498b-ac6f-6b844a5e4acf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:56.310331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8929a2d3-86a8-498b-ac6f-6b844a5e4acf admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:56.321706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1280/2549] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:56.325831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bb192a-67d6-4208-94a1-6f4246f86bc7 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:56.335784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bb192a-67d6-4208-94a1-6f4246f86bc7 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:56.340472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dbb8b8-2c4e-4cd4-81db-b75fd37aa660 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:56.341052 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dbb8b8-2c4e-4cd4-81db-b75fd37aa660 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:56.341329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dbb8b8-2c4e-4cd4-81db-b75fd37aa660 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:56.341601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dbb8b8-2c4e-4cd4-81db-b75fd37aa660 admin admin] PATH_INFO: `/v3/projects/680ff02ae4b14665b0e38564d46f0045/users/24ecf3c58b6c41ce96a7eed28f9a2e8f/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:56.343823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:56.357290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dbb8b8-2c4e-4cd4-81db-b75fd37aa660 admin admin] RBAC: Authorizing `identity:create_grant(project_id=680ff02ae4b14665b0e38564d46f0045, user_id=24ecf3c58b6c41ce96a7eed28f9a2e8f, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:56.359984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dbb8b8-2c4e-4cd4-81db-b75fd37aa660 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:56.453647 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1270/2550] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:56 2022] PUT /identity/v3/projects/680ff02ae4b14665b0e38564d46f0045/users/24ecf3c58b6c41ce96a7eed28f9a2e8f/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:56.465053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9ec56e-a229-4c9b-9a03-e5256042347d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:56.465336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9ec56e-a229-4c9b-9a03-e5256042347d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:56.465610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9ec56e-a229-4c9b-9a03-e5256042347d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:56.495216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5c9ec56e-a229-4c9b-9a03-e5256042347d None None] MFA Rules not processed for user `24ecf3c58b6c41ce96a7eed28f9a2e8f`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:56.496518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5c9ec56e-a229-4c9b-9a03-e5256042347d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:56.527383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:56.561147 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1281/2551] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:56 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:56.573050 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dbb8b8-2c4e-4cd4-81db-b75fd37aa660 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:56.574846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-36dbb8b8-2c4e-4cd4-81db-b75fd37aa660 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:56.633777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dbb8b8-2c4e-4cd4-81db-b75fd37aa660 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:56.637739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd30ede6-ebfa-489f-be55-2d8bdad02dca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:56.638380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd30ede6-ebfa-489f-be55-2d8bdad02dca service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:56.638656 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd30ede6-ebfa-489f-be55-2d8bdad02dca service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:56.638924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd30ede6-ebfa-489f-be55-2d8bdad02dca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:56.649510 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd30ede6-ebfa-489f-be55-2d8bdad02dca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:56.651295 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd30ede6-ebfa-489f-be55-2d8bdad02dca service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:56.662402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1271/2552] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:56 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:56.683212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9ec56e-a229-4c9b-9a03-e5256042347d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:56.747618 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9ec56e-a229-4c9b-9a03-e5256042347d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:56.752547 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:56.753175 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:56.753455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:56.753725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] PATH_INFO: `/v3/users/24ecf3c58b6c41ce96a7eed28f9a2e8f` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:56.755581 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] RBAC: Authorizing `identity:delete_user(user_id=24ecf3c58b6c41ce96a7eed28f9a2e8f)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:56.757683 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:56.947385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24ecf3c58b6c41ce96a7eed28f9a2e8f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:56.948032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24ecf3c58b6c41ce96a7eed28f9a2e8f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:56.955919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24ecf3c58b6c41ce96a7eed28f9a2e8f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:56.956679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24ecf3c58b6c41ce96a7eed28f9a2e8f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:57.035121 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24ecf3c58b6c41ce96a7eed28f9a2e8f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:57.054011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1282/2553] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:57:56 2022] DELETE /identity/v3/users/24ecf3c58b6c41ce96a7eed28f9a2e8f => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:57:57.058513 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6085b55-0314-4692-8f15-fb50ef288d5c None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:57.058802 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6085b55-0314-4692-8f15-fb50ef288d5c None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:57.059078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6085b55-0314-4692-8f15-fb50ef288d5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:57.079262 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f6085b55-0314-4692-8f15-fb50ef288d5c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:57.080469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f6085b55-0314-4692-8f15-fb50ef288d5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:57.140151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1272/2554] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:57:57.150012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:57.151468 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:57.209834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731b6c46-94d8-47b2-be9e-43f508bc6803 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:57.213393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378ddf42-849f-49a3-ace9-bcbf71eb6b8b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:57.213996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378ddf42-849f-49a3-ace9-bcbf71eb6b8b service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:57.214307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378ddf42-849f-49a3-ace9-bcbf71eb6b8b service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:57.214582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378ddf42-849f-49a3-ace9-bcbf71eb6b8b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:57.220566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:57.224936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378ddf42-849f-49a3-ace9-bcbf71eb6b8b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:57.226423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378ddf42-849f-49a3-ace9-bcbf71eb6b8b service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:57.236060 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1283/2555] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:57:57.659728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6085b55-0314-4692-8f15-fb50ef288d5c None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:57.661214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f6085b55-0314-4692-8f15-fb50ef288d5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:57.721034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6085b55-0314-4692-8f15-fb50ef288d5c None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:57.721034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:57.721034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:57.721034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:57.721778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b admin admin] PATH_INFO: `/v3/projects/680ff02ae4b14665b0e38564d46f0045` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:57.721778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b admin admin] RBAC: Authorizing `identity:delete_project(project_id=680ff02ae4b14665b0e38564d46f0045)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:57.721778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:57.847433 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '680ff02ae4b14665b0e38564d46f0045', 'request_id': 'req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:57.853698 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '680ff02ae4b14665b0e38564d46f0045', 'request_id': 'req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:57.859908 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '680ff02ae4b14665b0e38564d46f0045', 'request_id': 'req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:57.971992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b admin admin] The token cache is being invalidate because project 680ff02ae4b14665b0e38564d46f0045 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:57:57.972282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:57.972942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9b63ddd9-52ac-418f-aeb4-1a6945a6710b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:57:57.974476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1273/2556] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:57:57 2022] DELETE /identity/v3/projects/680ff02ae4b14665b0e38564d46f0045 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:57.980467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e28c2a-234f-4aca-9269-c9215e6a8d93 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:57.980925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e28c2a-234f-4aca-9269-c9215e6a8d93 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:57.981077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e28c2a-234f-4aca-9269-c9215e6a8d93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:58.001617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e0e28c2a-234f-4aca-9269-c9215e6a8d93 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:58.005167 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e0e28c2a-234f-4aca-9269-c9215e6a8d93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:58.064871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1284/2557] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:58.072795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7045270-f7e9-464c-8b45-c24d727a8166 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:58.073156 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7045270-f7e9-464c-8b45-c24d727a8166 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:58.073500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7045270-f7e9-464c-8b45-c24d727a8166 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:58.093195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-b7045270-f7e9-464c-8b45-c24d727a8166 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:58.094443 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b7045270-f7e9-464c-8b45-c24d727a8166 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:58.103364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1274/2558] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:57:58.107609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e28c2a-234f-4aca-9269-c9215e6a8d93 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:58.118396 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e28c2a-234f-4aca-9269-c9215e6a8d93 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:58.123336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c4e9c-587f-4403-89a4-46368f1d6a4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:58.123928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c4e9c-587f-4403-89a4-46368f1d6a4c admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:58.124209 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c4e9c-587f-4403-89a4-46368f1d6a4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:58.124481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c4e9c-587f-4403-89a4-46368f1d6a4c admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:58.125128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c4e9c-587f-4403-89a4-46368f1d6a4c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:57:58.125461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c4e9c-587f-4403-89a4-46368f1d6a4c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:58.127130 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c4e9c-587f-4403-89a4-46368f1d6a4c admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:58.138958 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1285/2559] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:57:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:57:58.143248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7045270-f7e9-464c-8b45-c24d727a8166 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:58.147701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:58.152939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7045270-f7e9-464c-8b45-c24d727a8166 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:58.157811 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bab57cc-8123-4d26-8cf8-6a8ad6beab9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:58.158429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bab57cc-8123-4d26-8cf8-6a8ad6beab9f admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:58.158708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bab57cc-8123-4d26-8cf8-6a8ad6beab9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:58.158975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bab57cc-8123-4d26-8cf8-6a8ad6beab9f admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:58.159676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bab57cc-8123-4d26-8cf8-6a8ad6beab9f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:58.161570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bab57cc-8123-4d26-8cf8-6a8ad6beab9f admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:58.242926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1275/2560] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:57:58 2022] POST /identity/v3/projects => generated 391 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:57:58.247174 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c4e9c-587f-4403-89a4-46368f1d6a4c admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:58.257956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c4e9c-587f-4403-89a4-46368f1d6a4c admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:58.315637 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10508dc-dff6-467d-8997-817da64ff4b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:58.316259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10508dc-dff6-467d-8997-817da64ff4b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:58.316539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10508dc-dff6-467d-8997-817da64ff4b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:58.316829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10508dc-dff6-467d-8997-817da64ff4b4 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:58.317447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10508dc-dff6-467d-8997-817da64ff4b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:58.319467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10508dc-dff6-467d-8997-817da64ff4b4 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:58.386592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1286/2561] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:57:58 2022] POST /identity/v3/users => generated 296 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:57:58.388512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bab57cc-8123-4d26-8cf8-6a8ad6beab9f admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:58.399624 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bab57cc-8123-4d26-8cf8-6a8ad6beab9f admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:58.404384 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7602a7eb-03d9-4af6-a98c-69d9a36e2b0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:58.405036 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7602a7eb-03d9-4af6-a98c-69d9a36e2b0f admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:58.405350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7602a7eb-03d9-4af6-a98c-69d9a36e2b0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:58.405627 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7602a7eb-03d9-4af6-a98c-69d9a36e2b0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:58.406318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7602a7eb-03d9-4af6-a98c-69d9a36e2b0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:58.407946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7602a7eb-03d9-4af6-a98c-69d9a36e2b0f admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:58.421019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1276/2562] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:58 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:58.425216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10508dc-dff6-467d-8997-817da64ff4b4 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:58.435642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10508dc-dff6-467d-8997-817da64ff4b4 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:58.440310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5179985d-2ac1-4cb3-b3ec-e6d6ec963f0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:58.440895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5179985d-2ac1-4cb3-b3ec-e6d6ec963f0e admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:58.441171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5179985d-2ac1-4cb3-b3ec-e6d6ec963f0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:58.441440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5179985d-2ac1-4cb3-b3ec-e6d6ec963f0e admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:58.442088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5179985d-2ac1-4cb3-b3ec-e6d6ec963f0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:58.443899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5179985d-2ac1-4cb3-b3ec-e6d6ec963f0e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:58.454610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1287/2563] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:57:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:57:58.458695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7602a7eb-03d9-4af6-a98c-69d9a36e2b0f admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:58.469385 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7602a7eb-03d9-4af6-a98c-69d9a36e2b0f admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:58.474020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252e0527-c490-4687-bdd7-77b381d73bae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:58.474729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252e0527-c490-4687-bdd7-77b381d73bae admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:58.475781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252e0527-c490-4687-bdd7-77b381d73bae admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:58.476062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252e0527-c490-4687-bdd7-77b381d73bae admin admin] PATH_INFO: `/v3/projects/7bcc22cf934d44c9861f857c9f2850b3/users/38d945b65d3a4ad1ab1ddf868767f781/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:58.491185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252e0527-c490-4687-bdd7-77b381d73bae admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bcc22cf934d44c9861f857c9f2850b3, user_id=38d945b65d3a4ad1ab1ddf868767f781, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:58.493955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252e0527-c490-4687-bdd7-77b381d73bae admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:58.590949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1277/2564] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:57:58 2022] PUT /identity/v3/projects/7bcc22cf934d44c9861f857c9f2850b3/users/38d945b65d3a4ad1ab1ddf868767f781/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:57:58.602666 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507690be-2210-4bb3-9c5e-89c268475140 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:58.603234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507690be-2210-4bb3-9c5e-89c268475140 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:58.603541 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507690be-2210-4bb3-9c5e-89c268475140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:58.633002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-507690be-2210-4bb3-9c5e-89c268475140 None None] MFA Rules not processed for user `38d945b65d3a4ad1ab1ddf868767f781`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:57:58.634262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-507690be-2210-4bb3-9c5e-89c268475140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:58.683164 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:57:58.703679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1288/2565] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:57:58 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:57:58.715364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252e0527-c490-4687-bdd7-77b381d73bae admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:57:58.717072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-252e0527-c490-4687-bdd7-77b381d73bae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:57:58.775605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252e0527-c490-4687-bdd7-77b381d73bae admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:57:58.779343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a09d446-122e-40c5-bf50-0e5c01b5c53c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:57:58.780054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a09d446-122e-40c5-bf50-0e5c01b5c53c service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:57:58.780329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a09d446-122e-40c5-bf50-0e5c01b5c53c service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:57:58.780596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a09d446-122e-40c5-bf50-0e5c01b5c53c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:57:58.789048 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a09d446-122e-40c5-bf50-0e5c01b5c53c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:57:58.790577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a09d446-122e-40c5-bf50-0e5c01b5c53c service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:57:58.799408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1278/2566] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:57:58 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:08.795326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=49,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:08.915348 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=5 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:15.371848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507690be-2210-4bb3-9c5e-89c268475140 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:15.373293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-507690be-2210-4bb3-9c5e-89c268475140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:15.432609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507690be-2210-4bb3-9c5e-89c268475140 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:15.436419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c90962-4e73-407e-b9db-98fc485db055 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:15.436900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c90962-4e73-407e-b9db-98fc485db055 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:15.437181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c90962-4e73-407e-b9db-98fc485db055 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:15.437459 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c90962-4e73-407e-b9db-98fc485db055 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:15.439433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-82c90962-4e73-407e-b9db-98fc485db055 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:15.510677 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c90962-4e73-407e-b9db-98fc485db055 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:15.512166 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c90962-4e73-407e-b9db-98fc485db055 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:15.523332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1289/2567] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:19.033332 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a09d446-122e-40c5-bf50-0e5c01b5c53c service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:19.034855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-2a09d446-122e-40c5-bf50-0e5c01b5c53c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:19.109115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a09d446-122e-40c5-bf50-0e5c01b5c53c service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:19.113709 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:19.114385 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:19.114628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:19.114897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] PATH_INFO: `/v3/users/107b596147ea42968db0be5330080a01` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:19.116051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] RBAC: Authorizing `identity:delete_user(user_id=107b596147ea42968db0be5330080a01)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:19.119060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:19.322672 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '107b596147ea42968db0be5330080a01'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:19.323830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '107b596147ea42968db0be5330080a01'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:19.331377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '107b596147ea42968db0be5330080a01'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:19.344532 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '107b596147ea42968db0be5330080a01'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:19.458520 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '107b596147ea42968db0be5330080a01'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:19.465417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1279/2568] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:19 2022] DELETE /identity/v3/users/107b596147ea42968db0be5330080a01 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:19.474250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37b0722-6821-487a-af60-6d90aea4c2ca None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:19.474719 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37b0722-6821-487a-af60-6d90aea4c2ca None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:19.474833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37b0722-6821-487a-af60-6d90aea4c2ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:19.495542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-f37b0722-6821-487a-af60-6d90aea4c2ca None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:19.496754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f37b0722-6821-487a-af60-6d90aea4c2ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:19.558064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1290/2569] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:19.567911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:19.569333 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:19.628955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613f4451-56aa-4271-929f-3c8c223c4f42 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:19.632696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3891260-b495-4d9d-b044-34906c4da10d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:19.633370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3891260-b495-4d9d-b044-34906c4da10d service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:19.633652 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3891260-b495-4d9d-b044-34906c4da10d service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:19.633926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3891260-b495-4d9d-b044-34906c4da10d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:19.643576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3891260-b495-4d9d-b044-34906c4da10d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:19.645050 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3891260-b495-4d9d-b044-34906c4da10d service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:19.655677 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1280/2570] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:20.140987 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37b0722-6821-487a-af60-6d90aea4c2ca None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:20.142841 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-f37b0722-6821-487a-af60-6d90aea4c2ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:20.164024 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37b0722-6821-487a-af60-6d90aea4c2ca None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:20.171916 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:20.172901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:20.173230 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:20.173547 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] PATH_INFO: `/v3/projects/53090206d87c4d9abbfed95ca4832a67` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:20.175125 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] RBAC: Authorizing `identity:delete_project(project_id=53090206d87c4d9abbfed95ca4832a67)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:20.178190 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:20.278700 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:20.286810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53090206d87c4d9abbfed95ca4832a67', 'request_id': 'req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:20.292821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53090206d87c4d9abbfed95ca4832a67', 'request_id': 'req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:20.412369 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53090206d87c4d9abbfed95ca4832a67', 'request_id': 'req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:20.424934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] The token cache is being invalidate because project 53090206d87c4d9abbfed95ca4832a67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:20.425111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:20.425670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:20.427173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1291/2571] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:20 2022] DELETE /identity/v3/projects/53090206d87c4d9abbfed95ca4832a67 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:20.431838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3891260-b495-4d9d-b044-34906c4da10d service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:20.441902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3891260-b495-4d9d-b044-34906c4da10d service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:20.460435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:20.501941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74603d82-9ff4-49ae-895e-189a878fed1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:20.502488 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74603d82-9ff4-49ae-895e-189a878fed1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:20.502766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74603d82-9ff4-49ae-895e-189a878fed1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:20.503032 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74603d82-9ff4-49ae-895e-189a878fed1b admin admin] PATH_INFO: `/v3/users/9b6f91894b1c45379b8b39c80d21dd75` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:20.504153 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74603d82-9ff4-49ae-895e-189a878fed1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b6f91894b1c45379b8b39c80d21dd75)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:20.506039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74603d82-9ff4-49ae-895e-189a878fed1b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:20.723906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-74603d82-9ff4-49ae-895e-189a878fed1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b6f91894b1c45379b8b39c80d21dd75'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:20.724494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-74603d82-9ff4-49ae-895e-189a878fed1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b6f91894b1c45379b8b39c80d21dd75'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:20.732737 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-74603d82-9ff4-49ae-895e-189a878fed1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b6f91894b1c45379b8b39c80d21dd75'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:20.744393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-74603d82-9ff4-49ae-895e-189a878fed1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b6f91894b1c45379b8b39c80d21dd75'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:20.812528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-74603d82-9ff4-49ae-895e-189a878fed1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b6f91894b1c45379b8b39c80d21dd75'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:20.819760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1281/2572] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:20 2022] DELETE /identity/v3/users/9b6f91894b1c45379b8b39c80d21dd75 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:21.224525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:21.225959 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:21.240902 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3061819-fb2b-4b77-ac3b-9b13d0e2241e admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:21.301914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf47bf92-f753-481a-9d40-fce49f5dd006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:21.302516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf47bf92-f753-481a-9d40-fce49f5dd006 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:21.302818 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf47bf92-f753-481a-9d40-fce49f5dd006 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:21.303081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf47bf92-f753-481a-9d40-fce49f5dd006 admin admin] PATH_INFO: `/v3/projects/ac092cb6c213427e867a48c96bb68acb` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:21.304477 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf47bf92-f753-481a-9d40-fce49f5dd006 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac092cb6c213427e867a48c96bb68acb)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:21.306479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf47bf92-f753-481a-9d40-fce49f5dd006 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:21.461901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-bf47bf92-f753-481a-9d40-fce49f5dd006 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac092cb6c213427e867a48c96bb68acb', 'request_id': 'req-bf47bf92-f753-481a-9d40-fce49f5dd006'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:21.467635 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-bf47bf92-f753-481a-9d40-fce49f5dd006 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac092cb6c213427e867a48c96bb68acb', 'request_id': 'req-bf47bf92-f753-481a-9d40-fce49f5dd006'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:21.560989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-bf47bf92-f753-481a-9d40-fce49f5dd006 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac092cb6c213427e867a48c96bb68acb', 'request_id': 'req-bf47bf92-f753-481a-9d40-fce49f5dd006'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:21.573650 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-bf47bf92-f753-481a-9d40-fce49f5dd006 admin admin] The token cache is being invalidate because project ac092cb6c213427e867a48c96bb68acb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:21.573940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-bf47bf92-f753-481a-9d40-fce49f5dd006 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:21.574614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-bf47bf92-f753-481a-9d40-fce49f5dd006 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:21.576111 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1292/2573] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:21 2022] DELETE /identity/v3/projects/ac092cb6c213427e867a48c96bb68acb => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:21.581053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffda272-db6e-47c0-8950-3813c94cecd4 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:21.581659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffda272-db6e-47c0-8950-3813c94cecd4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:21.581659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffda272-db6e-47c0-8950-3813c94cecd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:21.604329 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-7ffda272-db6e-47c0-8950-3813c94cecd4 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:21.605540 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-7ffda272-db6e-47c0-8950-3813c94cecd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:21.666909 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1282/2574] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:21.675671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb21452-4763-4283-85e2-0a2b55a58756 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:21.675989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb21452-4763-4283-85e2-0a2b55a58756 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:21.676265 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb21452-4763-4283-85e2-0a2b55a58756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:21.697323 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-ebb21452-4763-4283-85e2-0a2b55a58756 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:21.698585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ebb21452-4763-4283-85e2-0a2b55a58756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:21.707048 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1293/2575] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:21.712514 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffda272-db6e-47c0-8950-3813c94cecd4 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:21.722812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffda272-db6e-47c0-8950-3813c94cecd4 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:21.728696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e7453d-0bb4-41ad-a8e5-c7d0b48185f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:21.729302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e7453d-0bb4-41ad-a8e5-c7d0b48185f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:21.729577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e7453d-0bb4-41ad-a8e5-c7d0b48185f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:21.729845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e7453d-0bb4-41ad-a8e5-c7d0b48185f2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:21.730553 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e7453d-0bb4-41ad-a8e5-c7d0b48185f2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:58:21.730848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e7453d-0bb4-41ad-a8e5-c7d0b48185f2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:21.731697 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb21452-4763-4283-85e2-0a2b55a58756 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:21.732434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e7453d-0bb4-41ad-a8e5-c7d0b48185f2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:21.733115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ebb21452-4763-4283-85e2-0a2b55a58756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:21.744648 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1283/2576] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:58:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:58:21.749083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e7453d-0bb4-41ad-a8e5-c7d0b48185f2 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:21.759584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e7453d-0bb4-41ad-a8e5-c7d0b48185f2 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:21.764464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305793ae-d7bb-45ed-8db4-60c45241bf86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:21.765105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305793ae-d7bb-45ed-8db4-60c45241bf86 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:21.765446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305793ae-d7bb-45ed-8db4-60c45241bf86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:21.765842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305793ae-d7bb-45ed-8db4-60c45241bf86 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:21.766471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305793ae-d7bb-45ed-8db4-60c45241bf86 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:21.768509 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305793ae-d7bb-45ed-8db4-60c45241bf86 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:21.792462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:21.800637 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb21452-4763-4283-85e2-0a2b55a58756 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:21.800637 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ef7d68-827c-4ba2-bc95-6cf882dc07c7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:21.801181 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ef7d68-827c-4ba2-bc95-6cf882dc07c7 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:21.801524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ef7d68-827c-4ba2-bc95-6cf882dc07c7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:21.801794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ef7d68-827c-4ba2-bc95-6cf882dc07c7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:21.803598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a6ef7d68-827c-4ba2-bc95-6cf882dc07c7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:21.866583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ef7d68-827c-4ba2-bc95-6cf882dc07c7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:21.868129 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ef7d68-827c-4ba2-bc95-6cf882dc07c7 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:21.878021 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1294/2577] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:21 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:58:21.897702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:21.908743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1284/2578] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:21 2022] POST /identity/v3/projects => generated 387 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:58:21.909964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ef7d68-827c-4ba2-bc95-6cf882dc07c7 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:21.923002 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ef7d68-827c-4ba2-bc95-6cf882dc07c7 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:21.981117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d56340c-3556-4743-a95a-a56969904914 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:21.981735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d56340c-3556-4743-a95a-a56969904914 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:21.982013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d56340c-3556-4743-a95a-a56969904914 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:21.982564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d56340c-3556-4743-a95a-a56969904914 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:21.983141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d56340c-3556-4743-a95a-a56969904914 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:21.985078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d56340c-3556-4743-a95a-a56969904914 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:22.092619 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1295/2579] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:21 2022] POST /identity/v3/users => generated 294 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:58:22.096746 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305793ae-d7bb-45ed-8db4-60c45241bf86 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:22.108743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305793ae-d7bb-45ed-8db4-60c45241bf86 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:22.113620 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c420c55-1bab-4014-830f-5180ee055123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:22.117549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c420c55-1bab-4014-830f-5180ee055123 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:22.117549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c420c55-1bab-4014-830f-5180ee055123 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:22.117549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c420c55-1bab-4014-830f-5180ee055123 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:22.117549 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c420c55-1bab-4014-830f-5180ee055123 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:22.117961 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c420c55-1bab-4014-830f-5180ee055123 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:22.128057 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1285/2580] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:22.131875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d56340c-3556-4743-a95a-a56969904914 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:22.143536 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d56340c-3556-4743-a95a-a56969904914 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:22.148883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cc3041-aaa5-4b0e-a962-a5f518b62bf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:22.149485 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cc3041-aaa5-4b0e-a962-a5f518b62bf4 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:22.149787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cc3041-aaa5-4b0e-a962-a5f518b62bf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:22.150135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cc3041-aaa5-4b0e-a962-a5f518b62bf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:22.150809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cc3041-aaa5-4b0e-a962-a5f518b62bf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:22.152437 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cc3041-aaa5-4b0e-a962-a5f518b62bf4 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:22.163077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1296/2581] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:22.168296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c420c55-1bab-4014-830f-5180ee055123 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:22.178143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c420c55-1bab-4014-830f-5180ee055123 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:22.183083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b23562-13b6-4a3d-b64a-1ee88c7b8bcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:22.183721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b23562-13b6-4a3d-b64a-1ee88c7b8bcb admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:22.183994 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b23562-13b6-4a3d-b64a-1ee88c7b8bcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:22.184260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b23562-13b6-4a3d-b64a-1ee88c7b8bcb admin admin] PATH_INFO: `/v3/projects/47642bfa80ec4d55b3171d7a865efa39/users/335f9d0abd564166945d43399b40ba5f/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:22.200210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b23562-13b6-4a3d-b64a-1ee88c7b8bcb admin admin] RBAC: Authorizing `identity:create_grant(project_id=47642bfa80ec4d55b3171d7a865efa39, user_id=335f9d0abd564166945d43399b40ba5f, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:22.202885 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b23562-13b6-4a3d-b64a-1ee88c7b8bcb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:22.282256 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1286/2582] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:22 2022] PUT /identity/v3/projects/47642bfa80ec4d55b3171d7a865efa39/users/335f9d0abd564166945d43399b40ba5f/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:22.293938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4a6c38-1eff-494a-bf2f-5e5c968d653d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:22.294564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4a6c38-1eff-494a-bf2f-5e5c968d653d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:22.294564 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4a6c38-1eff-494a-bf2f-5e5c968d653d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:22.323235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-fe4a6c38-1eff-494a-bf2f-5e5c968d653d None None] MFA Rules not processed for user `335f9d0abd564166945d43399b40ba5f`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:22.324702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-fe4a6c38-1eff-494a-bf2f-5e5c968d653d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:22.392136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1297/2583] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:22 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:22.396848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b23562-13b6-4a3d-b64a-1ee88c7b8bcb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:22.487938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b23562-13b6-4a3d-b64a-1ee88c7b8bcb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:22.493507 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428dee7f-831a-4813-a8b2-e2bc479e6bdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:22.494100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428dee7f-831a-4813-a8b2-e2bc479e6bdf admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:22.494457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428dee7f-831a-4813-a8b2-e2bc479e6bdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:22.494729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428dee7f-831a-4813-a8b2-e2bc479e6bdf admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:22.497587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428dee7f-831a-4813-a8b2-e2bc479e6bdf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:22.497587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428dee7f-831a-4813-a8b2-e2bc479e6bdf admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:22.510265 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4a6c38-1eff-494a-bf2f-5e5c968d653d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:22.512341 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-fe4a6c38-1eff-494a-bf2f-5e5c968d653d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:22.525795 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:22.569716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4a6c38-1eff-494a-bf2f-5e5c968d653d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:22.573386 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a508295-b557-4164-96ef-91968ad06b70 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:22.573984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a508295-b557-4164-96ef-91968ad06b70 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:22.574294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a508295-b557-4164-96ef-91968ad06b70 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:22.574566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a508295-b557-4164-96ef-91968ad06b70 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:22.576373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6a508295-b557-4164-96ef-91968ad06b70 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:22.589462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a508295-b557-4164-96ef-91968ad06b70 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:22.593194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a508295-b557-4164-96ef-91968ad06b70 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:22.608755 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1287/2584] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:22 2022] POST /identity/v3/projects => generated 387 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:58:22.609785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428dee7f-831a-4813-a8b2-e2bc479e6bdf admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:22.613170 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1298/2585] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:58:22.684333 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428dee7f-831a-4813-a8b2-e2bc479e6bdf admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:22.689177 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1259fe8-973a-4bc7-8036-1f1be8d2de11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:22.689781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1259fe8-973a-4bc7-8036-1f1be8d2de11 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:22.690068 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1259fe8-973a-4bc7-8036-1f1be8d2de11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:22.690705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1259fe8-973a-4bc7-8036-1f1be8d2de11 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:22.690978 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1259fe8-973a-4bc7-8036-1f1be8d2de11 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:22.692889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1259fe8-973a-4bc7-8036-1f1be8d2de11 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:22.772966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1288/2586] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:22 2022] POST /identity/v3/users => generated 300 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:22.776999 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a508295-b557-4164-96ef-91968ad06b70 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:22.787104 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a508295-b557-4164-96ef-91968ad06b70 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:22.792675 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47952b2a-5908-4a12-a57d-b90397beb965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:22.793303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47952b2a-5908-4a12-a57d-b90397beb965 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:22.793583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47952b2a-5908-4a12-a57d-b90397beb965 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:22.793856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47952b2a-5908-4a12-a57d-b90397beb965 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:22.794534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47952b2a-5908-4a12-a57d-b90397beb965 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:22.796140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47952b2a-5908-4a12-a57d-b90397beb965 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:22.807225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1299/2587] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:22.812984 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1259fe8-973a-4bc7-8036-1f1be8d2de11 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:22.823356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1259fe8-973a-4bc7-8036-1f1be8d2de11 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:22.830214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a02c40b-24e0-4d4d-b253-25cdc18d24fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:22.831006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a02c40b-24e0-4d4d-b253-25cdc18d24fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:22.831350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a02c40b-24e0-4d4d-b253-25cdc18d24fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:22.832095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a02c40b-24e0-4d4d-b253-25cdc18d24fd admin admin] PATH_INFO: `/v3/projects/166cac64dc614f009dcb46432c2caf67/users/1f7221c263f14927bb07e8ca2cd7337b/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:22.853845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a02c40b-24e0-4d4d-b253-25cdc18d24fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=166cac64dc614f009dcb46432c2caf67, user_id=1f7221c263f14927bb07e8ca2cd7337b, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:22.856585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a02c40b-24e0-4d4d-b253-25cdc18d24fd admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:22.941525 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1289/2588] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:22 2022] PUT /identity/v3/projects/166cac64dc614f009dcb46432c2caf67/users/1f7221c263f14927bb07e8ca2cd7337b/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:22.942810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47952b2a-5908-4a12-a57d-b90397beb965 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:22.954040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47952b2a-5908-4a12-a57d-b90397beb965 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:23.015004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cdb2ea-cd0e-4227-8a86-3499fa6f34a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:23.015610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cdb2ea-cd0e-4227-8a86-3499fa6f34a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:23.015886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cdb2ea-cd0e-4227-8a86-3499fa6f34a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:23.016161 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cdb2ea-cd0e-4227-8a86-3499fa6f34a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:23.016802 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cdb2ea-cd0e-4227-8a86-3499fa6f34a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:23.018463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cdb2ea-cd0e-4227-8a86-3499fa6f34a3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:23.029315 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1300/2589] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:22 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:58:23.033082 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a02c40b-24e0-4d4d-b253-25cdc18d24fd admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:23.043769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a02c40b-24e0-4d4d-b253-25cdc18d24fd admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:23.048470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41057d11-6237-401d-991d-ab3785b60b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:23.049126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41057d11-6237-401d-991d-ab3785b60b88 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:23.049397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41057d11-6237-401d-991d-ab3785b60b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:23.049771 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41057d11-6237-401d-991d-ab3785b60b88 admin admin] PATH_INFO: `/v3/projects/166cac64dc614f009dcb46432c2caf67/users/1f7221c263f14927bb07e8ca2cd7337b/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:23.052078 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41057d11-6237-401d-991d-ab3785b60b88 admin admin] RBAC: Authorizing `identity:create_grant(project_id=166cac64dc614f009dcb46432c2caf67, user_id=1f7221c263f14927bb07e8ca2cd7337b, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:23.055728 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41057d11-6237-401d-991d-ab3785b60b88 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:23.063122 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:23.078559 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1290/2590] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:23 2022] PUT /identity/v3/projects/166cac64dc614f009dcb46432c2caf67/users/1f7221c263f14927bb07e8ca2cd7337b/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:23.080893 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58803fbe-da2f-4d5e-8701-84b24e9352b6 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:23.081538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58803fbe-da2f-4d5e-8701-84b24e9352b6 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:23.081664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58803fbe-da2f-4d5e-8701-84b24e9352b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:23.118389 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-58803fbe-da2f-4d5e-8701-84b24e9352b6 None None] MFA Rules not processed for user `1f7221c263f14927bb07e8ca2cd7337b`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:23.119658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-58803fbe-da2f-4d5e-8701-84b24e9352b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:23.198091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1301/2591] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:23 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:23.209448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41057d11-6237-401d-991d-ab3785b60b88 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:23.212037 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-41057d11-6237-401d-991d-ab3785b60b88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:23.276227 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41057d11-6237-401d-991d-ab3785b60b88 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:23.280029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7d3bf6-250b-4881-8c99-dd1a57e7507d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:23.280744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7d3bf6-250b-4881-8c99-dd1a57e7507d service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:23.281041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7d3bf6-250b-4881-8c99-dd1a57e7507d service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:23.281315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7d3bf6-250b-4881-8c99-dd1a57e7507d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:23.291955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7d3bf6-250b-4881-8c99-dd1a57e7507d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:23.293406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7d3bf6-250b-4881-8c99-dd1a57e7507d service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:23.303711 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1291/2592] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:23 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:23.481026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58803fbe-da2f-4d5e-8701-84b24e9352b6 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:23.508598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:23.545099 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58803fbe-da2f-4d5e-8701-84b24e9352b6 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:23.549808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:23.550497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:23.550779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:23.551051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] PATH_INFO: `/v3/users/335f9d0abd564166945d43399b40ba5f` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:23.552275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=335f9d0abd564166945d43399b40ba5f)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:23.554978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:23.701427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '335f9d0abd564166945d43399b40ba5f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:23.702265 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '335f9d0abd564166945d43399b40ba5f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:23.709849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '335f9d0abd564166945d43399b40ba5f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:23.711108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '335f9d0abd564166945d43399b40ba5f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:23.795432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '335f9d0abd564166945d43399b40ba5f'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:23.813397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1302/2593] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:23 2022] DELETE /identity/v3/users/335f9d0abd564166945d43399b40ba5f => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:23.818063 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b6f5d4-b60f-47b6-90bc-4fb5c466919d None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:23.818570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b6f5d4-b60f-47b6-90bc-4fb5c466919d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:23.818668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b6f5d4-b60f-47b6-90bc-4fb5c466919d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:23.838421 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d8b6f5d4-b60f-47b6-90bc-4fb5c466919d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:23.839730 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d8b6f5d4-b60f-47b6-90bc-4fb5c466919d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:23.914625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1292/2594] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:23.924907 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:23.927408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:23.997058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a78b347-1b11-4570-a93a-161308972c0b admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:24.002310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf12a5b-5fb4-4903-967a-f9bdce0ee105 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:24.002686 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf12a5b-5fb4-4903-967a-f9bdce0ee105 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:24.002989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf12a5b-5fb4-4903-967a-f9bdce0ee105 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:24.003259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf12a5b-5fb4-4903-967a-f9bdce0ee105 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:24.015080 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf12a5b-5fb4-4903-967a-f9bdce0ee105 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:24.016529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf12a5b-5fb4-4903-967a-f9bdce0ee105 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:24.027114 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1303/2595] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:24.452489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b6f5d4-b60f-47b6-90bc-4fb5c466919d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:24.454225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d8b6f5d4-b60f-47b6-90bc-4fb5c466919d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:24.468014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b6f5d4-b60f-47b6-90bc-4fb5c466919d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:24.473347 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:24.474965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:24.474965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:24.474965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] PATH_INFO: `/v3/projects/47642bfa80ec4d55b3171d7a865efa39` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:24.478608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=47642bfa80ec4d55b3171d7a865efa39)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:24.479485 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:24.621860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47642bfa80ec4d55b3171d7a865efa39', 'request_id': 'req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:24.627605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47642bfa80ec4d55b3171d7a865efa39', 'request_id': 'req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:24.635064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47642bfa80ec4d55b3171d7a865efa39', 'request_id': 'req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:24.720734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] The token cache is being invalidate because project 47642bfa80ec4d55b3171d7a865efa39 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:24.721022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:24.721766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:24.723722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1293/2596] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:24 2022] DELETE /identity/v3/projects/47642bfa80ec4d55b3171d7a865efa39 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:24.728244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf12a5b-5fb4-4903-967a-f9bdce0ee105 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:24.738660 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf12a5b-5fb4-4903-967a-f9bdce0ee105 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:24.802506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:24.838260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b294f3ce-47d4-4a3b-94a6-d2d4c1865572 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:24.838918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b294f3ce-47d4-4a3b-94a6-d2d4c1865572 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:24.839201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b294f3ce-47d4-4a3b-94a6-d2d4c1865572 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:24.839534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b294f3ce-47d4-4a3b-94a6-d2d4c1865572 admin admin] PATH_INFO: `/v3/users/1f7221c263f14927bb07e8ca2cd7337b` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:24.843282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b294f3ce-47d4-4a3b-94a6-d2d4c1865572 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f7221c263f14927bb07e8ca2cd7337b)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:24.846008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b294f3ce-47d4-4a3b-94a6-d2d4c1865572 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:25.000941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b294f3ce-47d4-4a3b-94a6-d2d4c1865572 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f7221c263f14927bb07e8ca2cd7337b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:25.001525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b294f3ce-47d4-4a3b-94a6-d2d4c1865572 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f7221c263f14927bb07e8ca2cd7337b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:25.009957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b294f3ce-47d4-4a3b-94a6-d2d4c1865572 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f7221c263f14927bb07e8ca2cd7337b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:25.010980 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b294f3ce-47d4-4a3b-94a6-d2d4c1865572 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f7221c263f14927bb07e8ca2cd7337b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:25.087711 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b294f3ce-47d4-4a3b-94a6-d2d4c1865572 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f7221c263f14927bb07e8ca2cd7337b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:25.108046 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1304/2597] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:24 2022] DELETE /identity/v3/users/1f7221c263f14927bb07e8ca2cd7337b => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:25.568019 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:25.569582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:25.582853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5a9419-850c-4c23-a6b2-6ed33721c6ac admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:25.612005 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:25.644379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d858e765-8ea9-422a-9ab5-bc100f096e73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:25.645033 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d858e765-8ea9-422a-9ab5-bc100f096e73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:25.645309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d858e765-8ea9-422a-9ab5-bc100f096e73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:25.645580 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d858e765-8ea9-422a-9ab5-bc100f096e73 admin admin] PATH_INFO: `/v3/projects/166cac64dc614f009dcb46432c2caf67` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:25.646731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d858e765-8ea9-422a-9ab5-bc100f096e73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=166cac64dc614f009dcb46432c2caf67)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:25.648671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d858e765-8ea9-422a-9ab5-bc100f096e73 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:25.745385 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d858e765-8ea9-422a-9ab5-bc100f096e73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '166cac64dc614f009dcb46432c2caf67', 'request_id': 'req-d858e765-8ea9-422a-9ab5-bc100f096e73'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:25.750676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d858e765-8ea9-422a-9ab5-bc100f096e73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '166cac64dc614f009dcb46432c2caf67', 'request_id': 'req-d858e765-8ea9-422a-9ab5-bc100f096e73'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:25.758509 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d858e765-8ea9-422a-9ab5-bc100f096e73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '166cac64dc614f009dcb46432c2caf67', 'request_id': 'req-d858e765-8ea9-422a-9ab5-bc100f096e73'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:25.852846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d858e765-8ea9-422a-9ab5-bc100f096e73 admin admin] The token cache is being invalidate because project 166cac64dc614f009dcb46432c2caf67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:25.853267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d858e765-8ea9-422a-9ab5-bc100f096e73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:25.853941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d858e765-8ea9-422a-9ab5-bc100f096e73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:25.855932 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1294/2598] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:25 2022] DELETE /identity/v3/projects/166cac64dc614f009dcb46432c2caf67 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:25.860914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dbd93c-560c-4a23-ad9a-88aafeee77cd None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:25.861377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dbd93c-560c-4a23-ad9a-88aafeee77cd None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:25.861630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dbd93c-560c-4a23-ad9a-88aafeee77cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:25.884525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e3dbd93c-560c-4a23-ad9a-88aafeee77cd None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:25.885884 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e3dbd93c-560c-4a23-ad9a-88aafeee77cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:25.963340 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1305/2599] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:25.972271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eadcf3-2475-41a9-b9e7-6a179135b324 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:25.972555 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eadcf3-2475-41a9-b9e7-6a179135b324 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:25.972830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eadcf3-2475-41a9-b9e7-6a179135b324 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:25.994343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-59eadcf3-2475-41a9-b9e7-6a179135b324 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:25.995571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-59eadcf3-2475-41a9-b9e7-6a179135b324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:26.011038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1295/2600] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:26.017366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dbd93c-560c-4a23-ad9a-88aafeee77cd None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:26.032429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dbd93c-560c-4a23-ad9a-88aafeee77cd None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:26.037720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef5f89c-3e04-46ed-b28c-51d918ea8540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:26.038568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef5f89c-3e04-46ed-b28c-51d918ea8540 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:26.038854 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef5f89c-3e04-46ed-b28c-51d918ea8540 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:26.039205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef5f89c-3e04-46ed-b28c-51d918ea8540 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:26.039968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef5f89c-3e04-46ed-b28c-51d918ea8540 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:58:26.040338 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef5f89c-3e04-46ed-b28c-51d918ea8540 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:26.042078 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef5f89c-3e04-46ed-b28c-51d918ea8540 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:26.056798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1306/2601] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:58:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:58:26.061319 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eadcf3-2475-41a9-b9e7-6a179135b324 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:26.072486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eadcf3-2475-41a9-b9e7-6a179135b324 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:26.078219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f764e7c-d83c-4eaf-80b2-d2c5fcd0b2c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:26.078814 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f764e7c-d83c-4eaf-80b2-d2c5fcd0b2c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:26.079682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f764e7c-d83c-4eaf-80b2-d2c5fcd0b2c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:26.079682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f764e7c-d83c-4eaf-80b2-d2c5fcd0b2c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:26.080382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f764e7c-d83c-4eaf-80b2-d2c5fcd0b2c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:26.082519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f764e7c-d83c-4eaf-80b2-d2c5fcd0b2c4 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:26.219708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1296/2602] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:26 2022] POST /identity/v3/projects => generated 385 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:58:26.221160 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef5f89c-3e04-46ed-b28c-51d918ea8540 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:26.233962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef5f89c-3e04-46ed-b28c-51d918ea8540 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:26.313172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:26.325522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe75fba-3d54-4d0c-a0d5-8884c4fb4b45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:26.326270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe75fba-3d54-4d0c-a0d5-8884c4fb4b45 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:26.326622 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe75fba-3d54-4d0c-a0d5-8884c4fb4b45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:26.326967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe75fba-3d54-4d0c-a0d5-8884c4fb4b45 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:26.327708 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe75fba-3d54-4d0c-a0d5-8884c4fb4b45 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:26.329849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe75fba-3d54-4d0c-a0d5-8884c4fb4b45 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:26.516318 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1307/2603] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:26 2022] POST /identity/v3/users => generated 293 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:26.521484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f764e7c-d83c-4eaf-80b2-d2c5fcd0b2c4 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:26.534451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f764e7c-d83c-4eaf-80b2-d2c5fcd0b2c4 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:26.539605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7904fdc-0ea4-466b-a62d-ea25256c9956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:26.540223 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7904fdc-0ea4-466b-a62d-ea25256c9956 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:26.540489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7904fdc-0ea4-466b-a62d-ea25256c9956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:26.540757 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7904fdc-0ea4-466b-a62d-ea25256c9956 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:26.543182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7904fdc-0ea4-466b-a62d-ea25256c9956 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:26.543182 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7904fdc-0ea4-466b-a62d-ea25256c9956 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:26.555724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1297/2604] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:26 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:26.562916 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe75fba-3d54-4d0c-a0d5-8884c4fb4b45 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:26.575256 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe75fba-3d54-4d0c-a0d5-8884c4fb4b45 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:26.581168 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0421924-626d-4924-b03d-211be4c2a55c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:26.581724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0421924-626d-4924-b03d-211be4c2a55c admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:26.582418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0421924-626d-4924-b03d-211be4c2a55c admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:26.582418 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0421924-626d-4924-b03d-211be4c2a55c admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:26.589703 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0421924-626d-4924-b03d-211be4c2a55c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:26.589703 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0421924-626d-4924-b03d-211be4c2a55c admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:26.596938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1308/2605] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:26 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:26.601410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7904fdc-0ea4-466b-a62d-ea25256c9956 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:26.612864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7904fdc-0ea4-466b-a62d-ea25256c9956 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:26.619449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb10baa-5166-408f-b0f3-7f386fcd7409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:26.620119 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb10baa-5166-408f-b0f3-7f386fcd7409 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:26.620407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb10baa-5166-408f-b0f3-7f386fcd7409 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:26.620676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb10baa-5166-408f-b0f3-7f386fcd7409 admin admin] PATH_INFO: `/v3/projects/2c414f4bff944e0180ce0c6aa4dffaf8/users/91768c61f62347688709ed89563194e8/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:26.637840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb10baa-5166-408f-b0f3-7f386fcd7409 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c414f4bff944e0180ce0c6aa4dffaf8, user_id=91768c61f62347688709ed89563194e8, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:26.642900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb10baa-5166-408f-b0f3-7f386fcd7409 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:26.700364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1298/2606] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:26 2022] PUT /identity/v3/projects/2c414f4bff944e0180ce0c6aa4dffaf8/users/91768c61f62347688709ed89563194e8/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:26.711997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f232ba7-5361-4f78-9e03-9e59df129b55 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:26.712278 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f232ba7-5361-4f78-9e03-9e59df129b55 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:26.712547 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f232ba7-5361-4f78-9e03-9e59df129b55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:26.750397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-3f232ba7-5361-4f78-9e03-9e59df129b55 None None] MFA Rules not processed for user `91768c61f62347688709ed89563194e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:26.751629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3f232ba7-5361-4f78-9e03-9e59df129b55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:26.839436 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1309/2607] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:26 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:26.844193 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb10baa-5166-408f-b0f3-7f386fcd7409 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:26.890748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f232ba7-5361-4f78-9e03-9e59df129b55 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:26.893164 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3f232ba7-5361-4f78-9e03-9e59df129b55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:26.927965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:26.946135 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb10baa-5166-408f-b0f3-7f386fcd7409 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:26.951352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203c68d7-0122-435d-96fb-e4ad74978d57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:26.955867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203c68d7-0122-435d-96fb-e4ad74978d57 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:26.955867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203c68d7-0122-435d-96fb-e4ad74978d57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:26.955867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203c68d7-0122-435d-96fb-e4ad74978d57 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:26.955867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203c68d7-0122-435d-96fb-e4ad74978d57 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:26.956221 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203c68d7-0122-435d-96fb-e4ad74978d57 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:26.980633 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f232ba7-5361-4f78-9e03-9e59df129b55 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:26.985993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:26.987038 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:26.987038 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:26.987356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] PATH_INFO: `/v3/users/bd8d967d9ecb410ea47e130ac3c8b4a6` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:27.001886 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd8d967d9ecb410ea47e130ac3c8b4a6)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:27.003903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:27.076147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1299/2608] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:26 2022] POST /identity/v3/projects => generated 387 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:27.080303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203c68d7-0122-435d-96fb-e4ad74978d57 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:27.142141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203c68d7-0122-435d-96fb-e4ad74978d57 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:27.146956 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f030e8-5428-4374-b2a2-92d634bf3ddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:27.147555 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f030e8-5428-4374-b2a2-92d634bf3ddd admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:27.147895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f030e8-5428-4374-b2a2-92d634bf3ddd admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:27.148315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f030e8-5428-4374-b2a2-92d634bf3ddd admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:27.148946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f030e8-5428-4374-b2a2-92d634bf3ddd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:27.150930 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f030e8-5428-4374-b2a2-92d634bf3ddd admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:27.204529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd8d967d9ecb410ea47e130ac3c8b4a6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.205262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd8d967d9ecb410ea47e130ac3c8b4a6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.209933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:27.212912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd8d967d9ecb410ea47e130ac3c8b4a6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.213598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd8d967d9ecb410ea47e130ac3c8b4a6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.240954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1300/2609] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:27 2022] POST /identity/v3/users => generated 294 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:58:27.245850 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f030e8-5428-4374-b2a2-92d634bf3ddd admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:27.247447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-23f030e8-5428-4374-b2a2-92d634bf3ddd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:27.260096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f030e8-5428-4374-b2a2-92d634bf3ddd admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:27.266328 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4390f629-c531-4cdb-a2f1-dbd3f5be5254 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:27.266925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4390f629-c531-4cdb-a2f1-dbd3f5be5254 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:27.267200 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4390f629-c531-4cdb-a2f1-dbd3f5be5254 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:27.267467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4390f629-c531-4cdb-a2f1-dbd3f5be5254 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:27.268105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4390f629-c531-4cdb-a2f1-dbd3f5be5254 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:27.269688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4390f629-c531-4cdb-a2f1-dbd3f5be5254 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:27.280498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1301/2610] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:27 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:58:27.284875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4390f629-c531-4cdb-a2f1-dbd3f5be5254 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:27.295449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4390f629-c531-4cdb-a2f1-dbd3f5be5254 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:27.300241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3468ee24-f1ce-4e0b-a5e4-4b7496709ec6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:27.300848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3468ee24-f1ce-4e0b-a5e4-4b7496709ec6 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:27.301147 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3468ee24-f1ce-4e0b-a5e4-4b7496709ec6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:27.301445 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3468ee24-f1ce-4e0b-a5e4-4b7496709ec6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:27.302086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3468ee24-f1ce-4e0b-a5e4-4b7496709ec6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:27.303697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3468ee24-f1ce-4e0b-a5e4-4b7496709ec6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:27.309116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd8d967d9ecb410ea47e130ac3c8b4a6'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.315615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1302/2611] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:58:27.320588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3468ee24-f1ce-4e0b-a5e4-4b7496709ec6 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:27.329253 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1310/2612] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:26 2022] DELETE /identity/v3/users/bd8d967d9ecb410ea47e130ac3c8b4a6 => generated 0 bytes in 439 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:27.330466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:27.330612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3468ee24-f1ce-4e0b-a5e4-4b7496709ec6 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:27.332462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:27.414575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346ebde0-6169-45f2-a9c6-971451d2f073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:27.414764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346ebde0-6169-45f2-a9c6-971451d2f073 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:27.415107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346ebde0-6169-45f2-a9c6-971451d2f073 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:27.415529 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346ebde0-6169-45f2-a9c6-971451d2f073 admin admin] PATH_INFO: `/v3/projects/baeb5149de2f4caeb0f8ba621ea37885/users/9bfb2f4f49dd42ebaa5fe6d9067da27f/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:27.419897 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c7df12-4f33-4ce9-9100-951487c7c21f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:27.425095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:27.425726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:27.426001 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:27.426446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] PATH_INFO: `/v3/users/38d945b65d3a4ad1ab1ddf868767f781` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:27.427644 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] RBAC: Authorizing `identity:delete_user(user_id=38d945b65d3a4ad1ab1ddf868767f781)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:27.430272 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:27.434619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346ebde0-6169-45f2-a9c6-971451d2f073 admin admin] RBAC: Authorizing `identity:create_grant(project_id=baeb5149de2f4caeb0f8ba621ea37885, user_id=9bfb2f4f49dd42ebaa5fe6d9067da27f, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:27.437777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346ebde0-6169-45f2-a9c6-971451d2f073 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:27.529628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1303/2613] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:27 2022] PUT /identity/v3/projects/baeb5149de2f4caeb0f8ba621ea37885/users/9bfb2f4f49dd42ebaa5fe6d9067da27f/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:27.539838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea56f4ca-a9fe-4be0-a523-91254c252315 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:27.540232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea56f4ca-a9fe-4be0-a523-91254c252315 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:27.540507 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea56f4ca-a9fe-4be0-a523-91254c252315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:27.569638 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-ea56f4ca-a9fe-4be0-a523-91254c252315 None None] MFA Rules not processed for user `9bfb2f4f49dd42ebaa5fe6d9067da27f`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:27.571197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ea56f4ca-a9fe-4be0-a523-91254c252315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:27.589615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:27.648858 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38d945b65d3a4ad1ab1ddf868767f781'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.650273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38d945b65d3a4ad1ab1ddf868767f781'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.650941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1304/2614] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:27 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:27.651857 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea56f4ca-a9fe-4be0-a523-91254c252315 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:27.656049 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ea56f4ca-a9fe-4be0-a523-91254c252315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:27.663267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38d945b65d3a4ad1ab1ddf868767f781'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.664140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38d945b65d3a4ad1ab1ddf868767f781'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.725315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea56f4ca-a9fe-4be0-a523-91254c252315 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:27.730828 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:27.732189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:27.732189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:27.732189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] PATH_INFO: `/v3/projects/ffc635aa12d14aa0b2880abe4e4f8f2f` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:27.743379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ffc635aa12d14aa0b2880abe4e4f8f2f)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:27.745317 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:27.772199 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38d945b65d3a4ad1ab1ddf868767f781'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.790567 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1311/2615] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:27 2022] DELETE /identity/v3/users/38d945b65d3a4ad1ab1ddf868767f781 => generated 0 bytes in 461 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:27.791427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:27.792764 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:27.841561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:27.856883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab55e7c-c2ef-4385-ae24-6780b1badc27 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:27.861661 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e6ee54-bc15-4552-8000-6e9ad49e5b1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:27.862243 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e6ee54-bc15-4552-8000-6e9ad49e5b1f admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:27.862508 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e6ee54-bc15-4552-8000-6e9ad49e5b1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:27.862833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e6ee54-bc15-4552-8000-6e9ad49e5b1f admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:27.863402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e6ee54-bc15-4552-8000-6e9ad49e5b1f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:27.865293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e6ee54-bc15-4552-8000-6e9ad49e5b1f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:27.876427 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ffc635aa12d14aa0b2880abe4e4f8f2f', 'request_id': 'req-da037faa-cc97-48b9-b893-63d5a25ef19b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.882134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ffc635aa12d14aa0b2880abe4e4f8f2f', 'request_id': 'req-da037faa-cc97-48b9-b893-63d5a25ef19b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.890394 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ffc635aa12d14aa0b2880abe4e4f8f2f', 'request_id': 'req-da037faa-cc97-48b9-b893-63d5a25ef19b'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.979281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1312/2616] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:27 2022] POST /identity/v3/projects => generated 387 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:27.980538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755862bf-d661-4369-80a3-20b247a1be87 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:27.980826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755862bf-d661-4369-80a3-20b247a1be87 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:27.981098 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755862bf-d661-4369-80a3-20b247a1be87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:27.990892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] The token cache is being invalidate because project ffc635aa12d14aa0b2880abe4e4f8f2f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:27.991206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.992582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:27.994060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1305/2617] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:27 2022] DELETE /identity/v3/projects/ffc635aa12d14aa0b2880abe4e4f8f2f => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:27.994915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:27.996367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:28.001455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-755862bf-d661-4369-80a3-20b247a1be87 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:28.002688 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-755862bf-d661-4369-80a3-20b247a1be87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:28.041533 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:28.063413 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1313/2618] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:28.064350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da037faa-cc97-48b9-b893-63d5a25ef19b admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.064658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d95dff-c58d-4af8-a0da-fca327fb44d8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.065008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d95dff-c58d-4af8-a0da-fca327fb44d8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.065200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d95dff-c58d-4af8-a0da-fca327fb44d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.069222 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e359900-2d40-45e2-bd03-ad34247cbf5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.069858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e359900-2d40-45e2-bd03-ad34247cbf5b admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.070141 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e359900-2d40-45e2-bd03-ad34247cbf5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.070446 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e359900-2d40-45e2-bd03-ad34247cbf5b admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.071079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e359900-2d40-45e2-bd03-ad34247cbf5b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.073042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e359900-2d40-45e2-bd03-ad34247cbf5b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.084603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-33d95dff-c58d-4af8-a0da-fca327fb44d8 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:28.085808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-33d95dff-c58d-4af8-a0da-fca327fb44d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:28.093823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1314/2619] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:28.094652 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d95dff-c58d-4af8-a0da-fca327fb44d8 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.095790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-33d95dff-c58d-4af8-a0da-fca327fb44d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:28.153520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d95dff-c58d-4af8-a0da-fca327fb44d8 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.157164 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100acb3b-5cbb-45f7-95c9-aece11408376 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.157874 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100acb3b-5cbb-45f7-95c9-aece11408376 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.158174 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100acb3b-5cbb-45f7-95c9-aece11408376 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.158533 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100acb3b-5cbb-45f7-95c9-aece11408376 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.169747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1306/2620] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:27 2022] POST /identity/v3/users => generated 300 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:28.169747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9790c2d7-7348-4eb5-af3e-15776baf5054 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.169747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9790c2d7-7348-4eb5-af3e-15776baf5054 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.169747 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9790c2d7-7348-4eb5-af3e-15776baf5054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.175173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100acb3b-5cbb-45f7-95c9-aece11408376 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.175173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100acb3b-5cbb-45f7-95c9-aece11408376 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.184487 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1315/2621] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:28.185304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100acb3b-5cbb-45f7-95c9-aece11408376 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.193054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-9790c2d7-7348-4eb5-af3e-15776baf5054 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:28.194292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-9790c2d7-7348-4eb5-af3e-15776baf5054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:28.195590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100acb3b-5cbb-45f7-95c9-aece11408376 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.200955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a142e5-fd45-4a5a-b5f9-b6f80c70fa5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.201666 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a142e5-fd45-4a5a-b5f9-b6f80c70fa5f admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.201828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a142e5-fd45-4a5a-b5f9-b6f80c70fa5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.202100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a142e5-fd45-4a5a-b5f9-b6f80c70fa5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.203342 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1307/2622] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:28.203483 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a142e5-fd45-4a5a-b5f9-b6f80c70fa5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.204400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a142e5-fd45-4a5a-b5f9-b6f80c70fa5f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.206958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9790c2d7-7348-4eb5-af3e-15776baf5054 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.219229 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1316/2623] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:28.219437 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9790c2d7-7348-4eb5-af3e-15776baf5054 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.220883 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a142e5-fd45-4a5a-b5f9-b6f80c70fa5f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.222435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a738b641-77c2-4083-aff8-c733db70c67c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.223105 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a738b641-77c2-4083-aff8-c733db70c67c admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.223385 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a738b641-77c2-4083-aff8-c733db70c67c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.223662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a738b641-77c2-4083-aff8-c733db70c67c admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.224453 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a738b641-77c2-4083-aff8-c733db70c67c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:58:28.224758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a738b641-77c2-4083-aff8-c733db70c67c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.227380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a738b641-77c2-4083-aff8-c733db70c67c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.233681 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a142e5-fd45-4a5a-b5f9-b6f80c70fa5f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.238933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0667bbe1-47ba-4076-b115-426cfe1f3fbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.239689 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0667bbe1-47ba-4076-b115-426cfe1f3fbb admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.243606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0667bbe1-47ba-4076-b115-426cfe1f3fbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.243752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1308/2624] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:58:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:58:28.243887 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0667bbe1-47ba-4076-b115-426cfe1f3fbb admin admin] PATH_INFO: `/v3/projects/fa9c08c0d4a44eb19d78a4c3c2bb5dd7/users/8db64d4edc1c4b11a9346a4c5cec65e2/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.245439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a738b641-77c2-4083-aff8-c733db70c67c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.260535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0667bbe1-47ba-4076-b115-426cfe1f3fbb admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa9c08c0d4a44eb19d78a4c3c2bb5dd7, user_id=8db64d4edc1c4b11a9346a4c5cec65e2, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.261051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a738b641-77c2-4083-aff8-c733db70c67c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.263901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0667bbe1-47ba-4076-b115-426cfe1f3fbb admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.266810 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43db6c4-f574-49e3-a851-4756eeaef468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.267213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43db6c4-f574-49e3-a851-4756eeaef468 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.267213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43db6c4-f574-49e3-a851-4756eeaef468 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.267503 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43db6c4-f574-49e3-a851-4756eeaef468 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.268066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43db6c4-f574-49e3-a851-4756eeaef468 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.270431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43db6c4-f574-49e3-a851-4756eeaef468 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.356731 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1317/2625] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:28 2022] PUT /identity/v3/projects/fa9c08c0d4a44eb19d78a4c3c2bb5dd7/users/8db64d4edc1c4b11a9346a4c5cec65e2/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:28.360521 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0667bbe1-47ba-4076-b115-426cfe1f3fbb admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.390015 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1309/2626] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:28 2022] POST /identity/v3/projects => generated 381 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:28.391153 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43db6c4-f574-49e3-a851-4756eeaef468 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.403984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:28.446805 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0667bbe1-47ba-4076-b115-426cfe1f3fbb admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.481659 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43db6c4-f574-49e3-a851-4756eeaef468 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.487445 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62e0fa9-98c1-4806-abe8-198848edf293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.489337 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62e0fa9-98c1-4806-abe8-198848edf293 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.489688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62e0fa9-98c1-4806-abe8-198848edf293 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.490034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62e0fa9-98c1-4806-abe8-198848edf293 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.490882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62e0fa9-98c1-4806-abe8-198848edf293 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.493645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62e0fa9-98c1-4806-abe8-198848edf293 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.509112 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11166bca-1289-4e06-9592-9e4fb4b86ece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.509728 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11166bca-1289-4e06-9592-9e4fb4b86ece admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.510393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11166bca-1289-4e06-9592-9e4fb4b86ece admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.510678 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11166bca-1289-4e06-9592-9e4fb4b86ece admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.511332 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11166bca-1289-4e06-9592-9e4fb4b86ece admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.512928 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11166bca-1289-4e06-9592-9e4fb4b86ece admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.522934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1318/2627] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:28 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:58:28.526966 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11166bca-1289-4e06-9592-9e4fb4b86ece admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.536535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11166bca-1289-4e06-9592-9e4fb4b86ece admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.542182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea6fe43-f945-4e48-8a39-b65cc7ce3992 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.542587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea6fe43-f945-4e48-8a39-b65cc7ce3992 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.542866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea6fe43-f945-4e48-8a39-b65cc7ce3992 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.543216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea6fe43-f945-4e48-8a39-b65cc7ce3992 admin admin] PATH_INFO: `/v3/projects/fa9c08c0d4a44eb19d78a4c3c2bb5dd7/users/8db64d4edc1c4b11a9346a4c5cec65e2/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.545384 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea6fe43-f945-4e48-8a39-b65cc7ce3992 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa9c08c0d4a44eb19d78a4c3c2bb5dd7, user_id=8db64d4edc1c4b11a9346a4c5cec65e2, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.548116 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea6fe43-f945-4e48-8a39-b65cc7ce3992 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.559176 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1319/2628] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:28 2022] PUT /identity/v3/projects/fa9c08c0d4a44eb19d78a4c3c2bb5dd7/users/8db64d4edc1c4b11a9346a4c5cec65e2/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:28.568989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69c310e-eeec-4f84-95f2-689eb047d257 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.569241 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69c310e-eeec-4f84-95f2-689eb047d257 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.575897 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69c310e-eeec-4f84-95f2-689eb047d257 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.605324 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c69c310e-eeec-4f84-95f2-689eb047d257 None None] MFA Rules not processed for user `8db64d4edc1c4b11a9346a4c5cec65e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:28.606655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c69c310e-eeec-4f84-95f2-689eb047d257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:28.631754 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1310/2629] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:28 2022] POST /identity/v3/users => generated 291 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:28.641420 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62e0fa9-98c1-4806-abe8-198848edf293 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.651521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62e0fa9-98c1-4806-abe8-198848edf293 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.688136 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1320/2630] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:28 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:28.689115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69c310e-eeec-4f84-95f2-689eb047d257 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.690680 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c69c310e-eeec-4f84-95f2-689eb047d257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:28.728650 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d2827d-1420-4fd6-8413-e5e2c1746dad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.729267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d2827d-1420-4fd6-8413-e5e2c1746dad admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.729541 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d2827d-1420-4fd6-8413-e5e2c1746dad admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.729809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d2827d-1420-4fd6-8413-e5e2c1746dad admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.730487 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d2827d-1420-4fd6-8413-e5e2c1746dad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.732178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d2827d-1420-4fd6-8413-e5e2c1746dad admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.745204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1311/2631] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:28 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:58:28.745204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d2827d-1420-4fd6-8413-e5e2c1746dad admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.745204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-b0d2827d-1420-4fd6-8413-e5e2c1746dad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:28.751841 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:28.759591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:28.765382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69c310e-eeec-4f84-95f2-689eb047d257 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.770462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b9edd6-fd34-47a7-bc52-669ccba2dd14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.770969 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9edd6-fd34-47a7-bc52-669ccba2dd14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.771236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9edd6-fd34-47a7-bc52-669ccba2dd14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.771510 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9edd6-fd34-47a7-bc52-669ccba2dd14 admin admin] PATH_INFO: `/v3/projects/7bcc22cf934d44c9861f857c9f2850b3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.772671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b9edd6-fd34-47a7-bc52-669ccba2dd14 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bcc22cf934d44c9861f857c9f2850b3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.775277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b9edd6-fd34-47a7-bc52-669ccba2dd14 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.804527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d2827d-1420-4fd6-8413-e5e2c1746dad admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.808240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef459b56-3fa8-48af-bb87-489a45b83668 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.808869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef459b56-3fa8-48af-bb87-489a45b83668 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.809195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef459b56-3fa8-48af-bb87-489a45b83668 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.809465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef459b56-3fa8-48af-bb87-489a45b83668 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.818498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef459b56-3fa8-48af-bb87-489a45b83668 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.820462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef459b56-3fa8-48af-bb87-489a45b83668 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.830142 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1312/2632] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:28 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:28.830949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef459b56-3fa8-48af-bb87-489a45b83668 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.843640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef459b56-3fa8-48af-bb87-489a45b83668 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.845992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ebd40d-9dd2-44dc-808f-160c78167dbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.846707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ebd40d-9dd2-44dc-808f-160c78167dbb admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.846992 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ebd40d-9dd2-44dc-808f-160c78167dbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.847261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ebd40d-9dd2-44dc-808f-160c78167dbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.847893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ebd40d-9dd2-44dc-808f-160c78167dbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.849528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ebd40d-9dd2-44dc-808f-160c78167dbb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.860442 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1313/2633] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:28.864578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ebd40d-9dd2-44dc-808f-160c78167dbb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:28.875418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ebd40d-9dd2-44dc-808f-160c78167dbb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:28.883799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482195c4-3170-4b86-95da-816558ecfeee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:28.883799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482195c4-3170-4b86-95da-816558ecfeee admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:28.883799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482195c4-3170-4b86-95da-816558ecfeee admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:28.883799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482195c4-3170-4b86-95da-816558ecfeee admin admin] PATH_INFO: `/v3/projects/d4112b8acbd7461ab66bad4b72a044ad/users/3f26f6f81ebb4ccbb2d3add3cf0e29e3/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:28.897594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482195c4-3170-4b86-95da-816558ecfeee admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4112b8acbd7461ab66bad4b72a044ad, user_id=3f26f6f81ebb4ccbb2d3add3cf0e29e3, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:28.900350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482195c4-3170-4b86-95da-816558ecfeee admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:28.996046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1314/2634] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:28 2022] PUT /identity/v3/projects/d4112b8acbd7461ab66bad4b72a044ad/users/3f26f6f81ebb4ccbb2d3add3cf0e29e3/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:29.003645 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-14b9edd6-fd34-47a7-bc52-669ccba2dd14 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bcc22cf934d44c9861f857c9f2850b3', 'request_id': 'req-14b9edd6-fd34-47a7-bc52-669ccba2dd14'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:29.007323 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c63d58-8e2f-4275-82fe-cb04da06ae40 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.008160 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c63d58-8e2f-4275-82fe-cb04da06ae40 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.008435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c63d58-8e2f-4275-82fe-cb04da06ae40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.010562 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-14b9edd6-fd34-47a7-bc52-669ccba2dd14 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bcc22cf934d44c9861f857c9f2850b3', 'request_id': 'req-14b9edd6-fd34-47a7-bc52-669ccba2dd14'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:29.036150 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-92c63d58-8e2f-4275-82fe-cb04da06ae40 None None] MFA Rules not processed for user `3f26f6f81ebb4ccbb2d3add3cf0e29e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:29.037366 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-92c63d58-8e2f-4275-82fe-cb04da06ae40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:29.096785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-14b9edd6-fd34-47a7-bc52-669ccba2dd14 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bcc22cf934d44c9861f857c9f2850b3', 'request_id': 'req-14b9edd6-fd34-47a7-bc52-669ccba2dd14'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:29.107883 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1315/2635] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:29 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:29.108996 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-14b9edd6-fd34-47a7-bc52-669ccba2dd14 admin admin] The token cache is being invalidate because project 7bcc22cf934d44c9861f857c9f2850b3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:29.109287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-14b9edd6-fd34-47a7-bc52-669ccba2dd14 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:29.109934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-14b9edd6-fd34-47a7-bc52-669ccba2dd14 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:29.111681 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1321/2636] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:28 2022] DELETE /identity/v3/projects/7bcc22cf934d44c9861f857c9f2850b3 => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:29.112695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c63d58-8e2f-4275-82fe-cb04da06ae40 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.116350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ba0e40-f931-46a3-bd31-6c56ac3df3c1 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.117172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ba0e40-f931-46a3-bd31-6c56ac3df3c1 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.117568 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ba0e40-f931-46a3-bd31-6c56ac3df3c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.137296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-92ba0e40-f931-46a3-bd31-6c56ac3df3c1 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:29.138545 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-92ba0e40-f931-46a3-bd31-6c56ac3df3c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:29.183543 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c63d58-8e2f-4275-82fe-cb04da06ae40 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.188502 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ce6535-c0af-4b2c-8c16-8e23f51de40e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.189111 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ce6535-c0af-4b2c-8c16-8e23f51de40e admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.189387 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ce6535-c0af-4b2c-8c16-8e23f51de40e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.189657 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ce6535-c0af-4b2c-8c16-8e23f51de40e admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.190597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ce6535-c0af-4b2c-8c16-8e23f51de40e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:29.192670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ce6535-c0af-4b2c-8c16-8e23f51de40e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:29.211634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1322/2637] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:29.220195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea4e30e-2e51-4d06-bfe8-cb3c0e58d61a None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.220475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea4e30e-2e51-4d06-bfe8-cb3c0e58d61a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.220917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea4e30e-2e51-4d06-bfe8-cb3c0e58d61a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.242638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-bea4e30e-2e51-4d06-bfe8-cb3c0e58d61a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:29.243837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-bea4e30e-2e51-4d06-bfe8-cb3c0e58d61a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:29.252654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1323/2638] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:29.258361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea4e30e-2e51-4d06-bfe8-cb3c0e58d61a None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.267730 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea4e30e-2e51-4d06-bfe8-cb3c0e58d61a None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.272625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dea6230-dfc0-4504-9a32-c93a900e305f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.273274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dea6230-dfc0-4504-9a32-c93a900e305f admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.273555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dea6230-dfc0-4504-9a32-c93a900e305f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.273828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dea6230-dfc0-4504-9a32-c93a900e305f admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.274506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dea6230-dfc0-4504-9a32-c93a900e305f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:58:29.274806 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dea6230-dfc0-4504-9a32-c93a900e305f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:29.276388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dea6230-dfc0-4504-9a32-c93a900e305f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:29.288044 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1324/2639] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:58:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:58:29.289212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1316/2640] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:29 2022] POST /identity/v3/projects => generated 381 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:58:29.292870 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dea6230-dfc0-4504-9a32-c93a900e305f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.295703 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ce6535-c0af-4b2c-8c16-8e23f51de40e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.297539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-13ce6535-c0af-4b2c-8c16-8e23f51de40e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:29.302915 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dea6230-dfc0-4504-9a32-c93a900e305f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.360304 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3375bd40-2e99-4e73-a85a-f7ac28b0e99e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.360913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3375bd40-2e99-4e73-a85a-f7ac28b0e99e admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.361194 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3375bd40-2e99-4e73-a85a-f7ac28b0e99e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.361484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3375bd40-2e99-4e73-a85a-f7ac28b0e99e admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.362104 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3375bd40-2e99-4e73-a85a-f7ac28b0e99e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:29.362904 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ce6535-c0af-4b2c-8c16-8e23f51de40e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.364035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3375bd40-2e99-4e73-a85a-f7ac28b0e99e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:29.368645 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ee7dca-a060-4e1c-89d9-e437a72066b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.369255 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ee7dca-a060-4e1c-89d9-e437a72066b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.369555 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ee7dca-a060-4e1c-89d9-e437a72066b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.369835 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ee7dca-a060-4e1c-89d9-e437a72066b5 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.370523 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ee7dca-a060-4e1c-89d9-e437a72066b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:29.371366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:29.372448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ee7dca-a060-4e1c-89d9-e437a72066b5 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:29.377573 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:29.475378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1317/2641] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:29 2022] POST /identity/v3/users => generated 297 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:29.480302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1325/2642] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:29 2022] POST /identity/v3/projects => generated 371 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:58:29.480924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ee7dca-a060-4e1c-89d9-e437a72066b5 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.481168 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3375bd40-2e99-4e73-a85a-f7ac28b0e99e admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.492500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ee7dca-a060-4e1c-89d9-e437a72066b5 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.543739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3375bd40-2e99-4e73-a85a-f7ac28b0e99e admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.548621 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c1c457-89e7-4ba6-98ad-55b8fb4ae7ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.549222 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c1c457-89e7-4ba6-98ad-55b8fb4ae7ba admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.549504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c1c457-89e7-4ba6-98ad-55b8fb4ae7ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.549605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bdf364-8732-4e9e-a5db-e2ad74d8a366 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.549782 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c1c457-89e7-4ba6-98ad-55b8fb4ae7ba admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.550176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bdf364-8732-4e9e-a5db-e2ad74d8a366 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.550449 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c1c457-89e7-4ba6-98ad-55b8fb4ae7ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:29.550550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bdf364-8732-4e9e-a5db-e2ad74d8a366 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.550725 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bdf364-8732-4e9e-a5db-e2ad74d8a366 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.551366 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1bdf364-8732-4e9e-a5db-e2ad74d8a366 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:29.552403 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c1c457-89e7-4ba6-98ad-55b8fb4ae7ba admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:29.552939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1bdf364-8732-4e9e-a5db-e2ad74d8a366 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:29.566995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1318/2643] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:29 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:58:29.567999 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bdf364-8732-4e9e-a5db-e2ad74d8a366 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.578088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bdf364-8732-4e9e-a5db-e2ad74d8a366 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.582778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f5fc2c-1a7a-4b33-9d33-cecb595069cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.583410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f5fc2c-1a7a-4b33-9d33-cecb595069cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.583688 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f5fc2c-1a7a-4b33-9d33-cecb595069cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.584064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f5fc2c-1a7a-4b33-9d33-cecb595069cf admin admin] PATH_INFO: `/v3/projects/c3a499db865b45dba98acd7bb8a306d3/users/c1d9c9cca74c42149cb79439db49d3ab/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.599466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f5fc2c-1a7a-4b33-9d33-cecb595069cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3a499db865b45dba98acd7bb8a306d3, user_id=c1d9c9cca74c42149cb79439db49d3ab, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:29.602878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f5fc2c-1a7a-4b33-9d33-cecb595069cf admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:29.662182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1326/2644] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:29 2022] POST /identity/v3/users => generated 286 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:29.666957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c1c457-89e7-4ba6-98ad-55b8fb4ae7ba admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.669691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1319/2645] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:29 2022] PUT /identity/v3/projects/c3a499db865b45dba98acd7bb8a306d3/users/c1d9c9cca74c42149cb79439db49d3ab/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:29.677946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f5fc2c-1a7a-4b33-9d33-cecb595069cf admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.682582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c1c457-89e7-4ba6-98ad-55b8fb4ae7ba admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.740869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc5597a-a304-4ed9-bf8a-0da43c81e40a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.741451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc5597a-a304-4ed9-bf8a-0da43c81e40a admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.741840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc5597a-a304-4ed9-bf8a-0da43c81e40a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.742343 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc5597a-a304-4ed9-bf8a-0da43c81e40a admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.742743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f5fc2c-1a7a-4b33-9d33-cecb595069cf admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.742990 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc5597a-a304-4ed9-bf8a-0da43c81e40a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:29.744608 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc5597a-a304-4ed9-bf8a-0da43c81e40a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:29.748304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5991cbb8-4e67-421e-b868-bfaf9a50a5c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.748901 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5991cbb8-4e67-421e-b868-bfaf9a50a5c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.749183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5991cbb8-4e67-421e-b868-bfaf9a50a5c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.749462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5991cbb8-4e67-421e-b868-bfaf9a50a5c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.750104 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5991cbb8-4e67-421e-b868-bfaf9a50a5c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:29.751773 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5991cbb8-4e67-421e-b868-bfaf9a50a5c9 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:29.754754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1327/2646] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:29 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:29.759093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc5597a-a304-4ed9-bf8a-0da43c81e40a admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.762572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1320/2647] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:29 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:29.766546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5991cbb8-4e67-421e-b868-bfaf9a50a5c9 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.769164 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc5597a-a304-4ed9-bf8a-0da43c81e40a admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.774787 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6e41dd-520e-41f4-9e48-6f00e738eb88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.775383 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e41dd-520e-41f4-9e48-6f00e738eb88 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.775664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e41dd-520e-41f4-9e48-6f00e738eb88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.775939 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e41dd-520e-41f4-9e48-6f00e738eb88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.776578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6e41dd-520e-41f4-9e48-6f00e738eb88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:29.776922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5991cbb8-4e67-421e-b868-bfaf9a50a5c9 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.778270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6e41dd-520e-41f4-9e48-6f00e738eb88 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:29.781839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c419342c-f800-4117-8ced-b3dacfe12d4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.782518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c419342c-f800-4117-8ced-b3dacfe12d4d admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.782795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c419342c-f800-4117-8ced-b3dacfe12d4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.783069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c419342c-f800-4117-8ced-b3dacfe12d4d admin admin] PATH_INFO: `/v3/projects/c3a499db865b45dba98acd7bb8a306d3/users/c1d9c9cca74c42149cb79439db49d3ab/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.785321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c419342c-f800-4117-8ced-b3dacfe12d4d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3a499db865b45dba98acd7bb8a306d3, user_id=c1d9c9cca74c42149cb79439db49d3ab, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:29.787764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c419342c-f800-4117-8ced-b3dacfe12d4d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:29.792823 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1328/2648] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:29.793864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6e41dd-520e-41f4-9e48-6f00e738eb88 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.799031 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1321/2649] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:29 2022] PUT /identity/v3/projects/c3a499db865b45dba98acd7bb8a306d3/users/c1d9c9cca74c42149cb79439db49d3ab/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:29.807358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6e41dd-520e-41f4-9e48-6f00e738eb88 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.809886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a15a7f-ea3c-4989-ba65-27af66eb5759 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.810206 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a15a7f-ea3c-4989-ba65-27af66eb5759 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.810494 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a15a7f-ea3c-4989-ba65-27af66eb5759 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.839153 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-75a15a7f-ea3c-4989-ba65-27af66eb5759 None None] MFA Rules not processed for user `c1d9c9cca74c42149cb79439db49d3ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:29.840457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-75a15a7f-ea3c-4989-ba65-27af66eb5759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:29.875190 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca47dc26-a04c-4f06-ada9-568278787fc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.875842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca47dc26-a04c-4f06-ada9-568278787fc5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.876120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca47dc26-a04c-4f06-ada9-568278787fc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.876393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca47dc26-a04c-4f06-ada9-568278787fc5 admin admin] PATH_INFO: `/v3/projects/7a6382d0a8d141e9ab64a41ae43b12af/users/6683479f4e1c419e950fa7497f194ac1/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.890901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca47dc26-a04c-4f06-ada9-568278787fc5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a6382d0a8d141e9ab64a41ae43b12af, user_id=6683479f4e1c419e950fa7497f194ac1, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:29.893598 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca47dc26-a04c-4f06-ada9-568278787fc5 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:29.914987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1322/2650] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:29 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:29.926845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a15a7f-ea3c-4989-ba65-27af66eb5759 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:29.928285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-75a15a7f-ea3c-4989-ba65-27af66eb5759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:29.988457 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a15a7f-ea3c-4989-ba65-27af66eb5759 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:29.992922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf57b46-f7b4-4d8b-a58d-8dfd8aa9da94 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:29.993547 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf57b46-f7b4-4d8b-a58d-8dfd8aa9da94 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:29.993840 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf57b46-f7b4-4d8b-a58d-8dfd8aa9da94 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:29.994107 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf57b46-f7b4-4d8b-a58d-8dfd8aa9da94 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:29.998729 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1329/2651] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:29 2022] PUT /identity/v3/projects/7a6382d0a8d141e9ab64a41ae43b12af/users/6683479f4e1c419e950fa7497f194ac1/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:30.004216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c924c1b1-a7cd-4c3d-ae6e-4dd95e8fca67 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:30.004788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c924c1b1-a7cd-4c3d-ae6e-4dd95e8fca67 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:30.004788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c924c1b1-a7cd-4c3d-ae6e-4dd95e8fca67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:30.005116 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf57b46-f7b4-4d8b-a58d-8dfd8aa9da94 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:30.006471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf57b46-f7b4-4d8b-a58d-8dfd8aa9da94 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:30.017763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1323/2652] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:29 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:30.026321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c924c1b1-a7cd-4c3d-ae6e-4dd95e8fca67 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:30.027545 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c924c1b1-a7cd-4c3d-ae6e-4dd95e8fca67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:30.052880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:30.068832 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf57b46-f7b4-4d8b-a58d-8dfd8aa9da94 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:30.070455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ecf57b46-f7b4-4d8b-a58d-8dfd8aa9da94 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:30.081611 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf57b46-f7b4-4d8b-a58d-8dfd8aa9da94 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:30.089869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1330/2653] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:30.099239 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c924c1b1-a7cd-4c3d-ae6e-4dd95e8fca67 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:30.100587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c924c1b1-a7cd-4c3d-ae6e-4dd95e8fca67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:30.140959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bacb445-f3e5-4e2b-a141-3558473af87d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:30.141562 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bacb445-f3e5-4e2b-a141-3558473af87d service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:30.141837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bacb445-f3e5-4e2b-a141-3558473af87d service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:30.142103 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bacb445-f3e5-4e2b-a141-3558473af87d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:30.144020 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6bacb445-f3e5-4e2b-a141-3558473af87d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:30.159205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c924c1b1-a7cd-4c3d-ae6e-4dd95e8fca67 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:30.162925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a8b1a0-b6b1-4755-ab41-20e14efb587e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:30.164169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a8b1a0-b6b1-4755-ab41-20e14efb587e service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:30.164446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a8b1a0-b6b1-4755-ab41-20e14efb587e service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:30.166805 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a8b1a0-b6b1-4755-ab41-20e14efb587e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:30.173872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a8b1a0-b6b1-4755-ab41-20e14efb587e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:30.175363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a8b1a0-b6b1-4755-ab41-20e14efb587e service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:30.186303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1331/2654] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:58:30.189749 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:30.202438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a8b1a0-b6b1-4755-ab41-20e14efb587e service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:30.206847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bacb445-f3e5-4e2b-a141-3558473af87d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:30.208649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bacb445-f3e5-4e2b-a141-3558473af87d service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:30.212081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a8b1a0-b6b1-4755-ab41-20e14efb587e service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:30.216518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7d0794-7a18-4ff8-aab7-4e7aba6f2a47 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:30.217119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7d0794-7a18-4ff8-aab7-4e7aba6f2a47 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:30.217400 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7d0794-7a18-4ff8-aab7-4e7aba6f2a47 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:30.217673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7d0794-7a18-4ff8-aab7-4e7aba6f2a47 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:30.218496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1324/2655] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:30 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:58:30.219478 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4f7d0794-7a18-4ff8-aab7-4e7aba6f2a47 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:30.282722 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7d0794-7a18-4ff8-aab7-4e7aba6f2a47 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:30.284267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7d0794-7a18-4ff8-aab7-4e7aba6f2a47 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:30.293735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1332/2656] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:30 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:30.452763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:31.878864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d58b5b-c4b3-4518-91f8-46eb8a70f5df None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:31.879527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d58b5b-c4b3-4518-91f8-46eb8a70f5df None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:31.879527 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d58b5b-c4b3-4518-91f8-46eb8a70f5df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:31.903030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-47d58b5b-c4b3-4518-91f8-46eb8a70f5df None None] MFA Rules not processed for user `71ff73cade8b42c7ae57be1e3ef0e028`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:31.904508 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-47d58b5b-c4b3-4518-91f8-46eb8a70f5df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:32.118782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1325/2657] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:31 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:32.191077 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7d0794-7a18-4ff8-aab7-4e7aba6f2a47 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:32.201801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7d0794-7a18-4ff8-aab7-4e7aba6f2a47 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:32.206102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4666af90-fb03-4e31-96e8-f1cab8539699 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:32.209058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4666af90-fb03-4e31-96e8-f1cab8539699 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:32.209058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4666af90-fb03-4e31-96e8-f1cab8539699 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:32.209058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4666af90-fb03-4e31-96e8-f1cab8539699 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:32.210373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4666af90-fb03-4e31-96e8-f1cab8539699 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:32.242296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:32.293466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4666af90-fb03-4e31-96e8-f1cab8539699 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:32.294953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4666af90-fb03-4e31-96e8-f1cab8539699 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:32.308127 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1333/2658] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:36.384292 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84bb05b-f2de-40ca-8a81-dea28afdab8a None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:36.384812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84bb05b-f2de-40ca-8a81-dea28afdab8a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:36.384812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84bb05b-f2de-40ca-8a81-dea28afdab8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:36.412941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d84bb05b-f2de-40ca-8a81-dea28afdab8a None None] MFA Rules not processed for user `6683479f4e1c419e950fa7497f194ac1`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:36.414224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d84bb05b-f2de-40ca-8a81-dea28afdab8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:36.480571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1326/2659] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:36 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:36.484992 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4666af90-fb03-4e31-96e8-f1cab8539699 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:36.496415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4666af90-fb03-4e31-96e8-f1cab8539699 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:36.501525 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efc9314-69e6-458b-a424-bbdece0ac389 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:36.502182 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efc9314-69e6-458b-a424-bbdece0ac389 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:36.502605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efc9314-69e6-458b-a424-bbdece0ac389 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:36.502899 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efc9314-69e6-458b-a424-bbdece0ac389 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:36.503520 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0efc9314-69e6-458b-a424-bbdece0ac389 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:36.505467 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0efc9314-69e6-458b-a424-bbdece0ac389 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:36.625964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1334/2660] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:36 2022] POST /identity/v3/projects => generated 373 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:36.630131 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84bb05b-f2de-40ca-8a81-dea28afdab8a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:36.641778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84bb05b-f2de-40ca-8a81-dea28afdab8a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:36.702632 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:36.709898 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83da4d8a-78db-4244-92a9-ab971379f456 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:36.722225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83da4d8a-78db-4244-92a9-ab971379f456 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:36.722225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83da4d8a-78db-4244-92a9-ab971379f456 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:36.722225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83da4d8a-78db-4244-92a9-ab971379f456 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:36.722225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83da4d8a-78db-4244-92a9-ab971379f456 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:36.722225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83da4d8a-78db-4244-92a9-ab971379f456 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:36.793051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1327/2661] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:36 2022] POST /identity/v3/users => generated 293 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:36.797070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efc9314-69e6-458b-a424-bbdece0ac389 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:36.808588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efc9314-69e6-458b-a424-bbdece0ac389 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:36.813682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b37122d-13b8-4b71-9aa6-f380410b8342 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:36.814769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b37122d-13b8-4b71-9aa6-f380410b8342 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:36.815119 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b37122d-13b8-4b71-9aa6-f380410b8342 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:36.815466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b37122d-13b8-4b71-9aa6-f380410b8342 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:36.816220 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b37122d-13b8-4b71-9aa6-f380410b8342 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:36.818052 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b37122d-13b8-4b71-9aa6-f380410b8342 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:36.831029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1335/2662] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:36 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:36.837176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83da4d8a-78db-4244-92a9-ab971379f456 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:36.848734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83da4d8a-78db-4244-92a9-ab971379f456 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:36.856316 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f69e7a0-0681-44bb-bfd7-75e970c899e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:36.856989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f69e7a0-0681-44bb-bfd7-75e970c899e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:36.857270 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f69e7a0-0681-44bb-bfd7-75e970c899e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:36.857537 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f69e7a0-0681-44bb-bfd7-75e970c899e3 admin admin] PATH_INFO: `/v3/projects/c98e33b751bf493781505f6c0e47cf33/users/cac8705e5db7411d8342e69c28f1fbe2/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:36.875017 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f69e7a0-0681-44bb-bfd7-75e970c899e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c98e33b751bf493781505f6c0e47cf33, user_id=cac8705e5db7411d8342e69c28f1fbe2, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:36.877724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f69e7a0-0681-44bb-bfd7-75e970c899e3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:36.944237 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1328/2663] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:36 2022] PUT /identity/v3/projects/c98e33b751bf493781505f6c0e47cf33/users/cac8705e5db7411d8342e69c28f1fbe2/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:36.948659 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b37122d-13b8-4b71-9aa6-f380410b8342 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:36.958859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b37122d-13b8-4b71-9aa6-f380410b8342 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:37.024865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785ca316-0465-4923-8fa2-4fe624ca64ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:37.025511 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785ca316-0465-4923-8fa2-4fe624ca64ad admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:37.025792 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785ca316-0465-4923-8fa2-4fe624ca64ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:37.026065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785ca316-0465-4923-8fa2-4fe624ca64ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:37.026833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785ca316-0465-4923-8fa2-4fe624ca64ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:37.028493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785ca316-0465-4923-8fa2-4fe624ca64ad admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:37.039733 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1336/2664] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:36 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:37.043950 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f69e7a0-0681-44bb-bfd7-75e970c899e3 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:37.055120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f69e7a0-0681-44bb-bfd7-75e970c899e3 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:37.061184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5713074-7939-4b50-b258-1c1b0f1e5942 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:37.061850 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5713074-7939-4b50-b258-1c1b0f1e5942 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:37.062126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5713074-7939-4b50-b258-1c1b0f1e5942 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:37.062502 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5713074-7939-4b50-b258-1c1b0f1e5942 admin admin] PATH_INFO: `/v3/projects/c98e33b751bf493781505f6c0e47cf33/users/cac8705e5db7411d8342e69c28f1fbe2/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:37.064553 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5713074-7939-4b50-b258-1c1b0f1e5942 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c98e33b751bf493781505f6c0e47cf33, user_id=cac8705e5db7411d8342e69c28f1fbe2, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:37.067261 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5713074-7939-4b50-b258-1c1b0f1e5942 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:37.078164 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1329/2665] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:37 2022] PUT /identity/v3/projects/c98e33b751bf493781505f6c0e47cf33/users/cac8705e5db7411d8342e69c28f1fbe2/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:39.148135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785ca316-0465-4923-8fa2-4fe624ca64ad admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:39.300638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785ca316-0465-4923-8fa2-4fe624ca64ad admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:39.340287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:39.361642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa1b111-340b-4491-b5c7-c3c23e47cb86 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:39.362398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa1b111-340b-4491-b5c7-c3c23e47cb86 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:39.362756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa1b111-340b-4491-b5c7-c3c23e47cb86 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:39.363103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa1b111-340b-4491-b5c7-c3c23e47cb86 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:39.366059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-2aa1b111-340b-4491-b5c7-c3c23e47cb86 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:39.462005 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa1b111-340b-4491-b5c7-c3c23e47cb86 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:39.464543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa1b111-340b-4491-b5c7-c3c23e47cb86 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:39.487373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1337/2666] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:39 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:44.137006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d203a124-ca25-475b-b76e-050c067fdb63 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:44.137569 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d203a124-ca25-475b-b76e-050c067fdb63 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:44.137707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d203a124-ca25-475b-b76e-050c067fdb63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:44.169364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d203a124-ca25-475b-b76e-050c067fdb63 None None] MFA Rules not processed for user `cac8705e5db7411d8342e69c28f1fbe2`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:44.170785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d203a124-ca25-475b-b76e-050c067fdb63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:44.255877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1330/2667] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:44 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:44.273231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa1b111-340b-4491-b5c7-c3c23e47cb86 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:44.276008 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-2aa1b111-340b-4491-b5c7-c3c23e47cb86 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:44.369680 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa1b111-340b-4491-b5c7-c3c23e47cb86 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:44.375851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699f1286-1c6f-410d-977c-a42f18705157 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:44.376504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699f1286-1c6f-410d-977c-a42f18705157 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:44.376785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699f1286-1c6f-410d-977c-a42f18705157 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:44.377051 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699f1286-1c6f-410d-977c-a42f18705157 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:44.388051 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d203a124-ca25-475b-b76e-050c067fdb63 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:44.459093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699f1286-1c6f-410d-977c-a42f18705157 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:44.461021 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699f1286-1c6f-410d-977c-a42f18705157 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:44.471871 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1338/2668] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:44 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:44.472774 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d203a124-ca25-475b-b76e-050c067fdb63 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:44.479391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921b202d-06f5-4c25-966d-f276286dbeb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:44.480239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921b202d-06f5-4c25-966d-f276286dbeb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:44.480707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921b202d-06f5-4c25-966d-f276286dbeb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:44.481145 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921b202d-06f5-4c25-966d-f276286dbeb9 admin admin] PATH_INFO: `/v3/users/3f26f6f81ebb4ccbb2d3add3cf0e29e3` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:44.482601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921b202d-06f5-4c25-966d-f276286dbeb9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f26f6f81ebb4ccbb2d3add3cf0e29e3)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:44.484790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921b202d-06f5-4c25-966d-f276286dbeb9 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:44.615338 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699f1286-1c6f-410d-977c-a42f18705157 service nova] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:44.617498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-699f1286-1c6f-410d-977c-a42f18705157 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:44.631012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699f1286-1c6f-410d-977c-a42f18705157 service nova] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:44.635589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c8dfb9-7c6a-4137-a707-836b9148c220 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:44.636207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c8dfb9-7c6a-4137-a707-836b9148c220 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:44.636486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c8dfb9-7c6a-4137-a707-836b9148c220 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:44.636755 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c8dfb9-7c6a-4137-a707-836b9148c220 admin admin] PATH_INFO: `/v3/users/91768c61f62347688709ed89563194e8` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:44.637911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c8dfb9-7c6a-4137-a707-836b9148c220 admin admin] RBAC: Authorizing `identity:delete_user(user_id=91768c61f62347688709ed89563194e8)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:44.639916 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c8dfb9-7c6a-4137-a707-836b9148c220 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:44.658101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:44.767045 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:44.777271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-921b202d-06f5-4c25-966d-f276286dbeb9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f26f6f81ebb4ccbb2d3add3cf0e29e3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:44.777934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-921b202d-06f5-4c25-966d-f276286dbeb9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f26f6f81ebb4ccbb2d3add3cf0e29e3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:44.786165 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-921b202d-06f5-4c25-966d-f276286dbeb9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f26f6f81ebb4ccbb2d3add3cf0e29e3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:44.805175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-921b202d-06f5-4c25-966d-f276286dbeb9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f26f6f81ebb4ccbb2d3add3cf0e29e3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:44.893612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-921b202d-06f5-4c25-966d-f276286dbeb9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f26f6f81ebb4ccbb2d3add3cf0e29e3'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:44.903478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1331/2669] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:44 2022] DELETE /identity/v3/users/3f26f6f81ebb4ccbb2d3add3cf0e29e3 => generated 0 bytes in 520 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:44.908057 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dcc25a-4b2d-4915-9e31-36304471ebb9 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:44.908364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dcc25a-4b2d-4915-9e31-36304471ebb9 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:44.908774 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dcc25a-4b2d-4915-9e31-36304471ebb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:44.915672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a6c8dfb9-7c6a-4137-a707-836b9148c220 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91768c61f62347688709ed89563194e8'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:44.915672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a6c8dfb9-7c6a-4137-a707-836b9148c220 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91768c61f62347688709ed89563194e8'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:44.923079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a6c8dfb9-7c6a-4137-a707-836b9148c220 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91768c61f62347688709ed89563194e8'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:44.923801 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a6c8dfb9-7c6a-4137-a707-836b9148c220 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91768c61f62347688709ed89563194e8'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:44.931007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-85dcc25a-4b2d-4915-9e31-36304471ebb9 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:44.932783 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-85dcc25a-4b2d-4915-9e31-36304471ebb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:45.017830 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1332/2670] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:45.024653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-a6c8dfb9-7c6a-4137-a707-836b9148c220 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91768c61f62347688709ed89563194e8'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:45.030374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dcc25a-4b2d-4915-9e31-36304471ebb9 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:45.032151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-85dcc25a-4b2d-4915-9e31-36304471ebb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:45.045438 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1339/2671] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:44 2022] DELETE /identity/v3/users/91768c61f62347688709ed89563194e8 => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:45.050845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7139941d-3686-4fac-a999-f831166d3418 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:45.050845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7139941d-3686-4fac-a999-f831166d3418 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:45.050845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7139941d-3686-4fac-a999-f831166d3418 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:45.079878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-7139941d-3686-4fac-a999-f831166d3418 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:45.079878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7139941d-3686-4fac-a999-f831166d3418 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:45.108517 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dcc25a-4b2d-4915-9e31-36304471ebb9 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:45.158927 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1340/2672] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:45.176464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0918478-c115-4a95-b24f-081479e93fdf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:45.176957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7139941d-3686-4fac-a999-f831166d3418 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:45.179389 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7139941d-3686-4fac-a999-f831166d3418 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:45.179539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0918478-c115-4a95-b24f-081479e93fdf service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:45.179891 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0918478-c115-4a95-b24f-081479e93fdf service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:45.180201 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0918478-c115-4a95-b24f-081479e93fdf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:45.197844 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0918478-c115-4a95-b24f-081479e93fdf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:45.197844 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0918478-c115-4a95-b24f-081479e93fdf service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:45.198238 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7139941d-3686-4fac-a999-f831166d3418 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:45.202211 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc43fd9-b167-4f9c-ab90-11d1000a1e9f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:45.202905 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc43fd9-b167-4f9c-ab90-11d1000a1e9f service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:45.203310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc43fd9-b167-4f9c-ab90-11d1000a1e9f service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:45.203640 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc43fd9-b167-4f9c-ab90-11d1000a1e9f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:45.207518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1333/2673] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:45.215809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc43fd9-b167-4f9c-ab90-11d1000a1e9f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:45.217410 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc43fd9-b167-4f9c-ab90-11d1000a1e9f service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:45.228909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1341/2674] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:58:45.743128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0918478-c115-4a95-b24f-081479e93fdf service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:45.744572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a0918478-c115-4a95-b24f-081479e93fdf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:45.766214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0918478-c115-4a95-b24f-081479e93fdf service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:45.776958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:45.776958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:45.776958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:45.776958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] PATH_INFO: `/v3/projects/2c414f4bff944e0180ce0c6aa4dffaf8` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:45.777759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2c414f4bff944e0180ce0c6aa4dffaf8)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:45.777759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:45.811461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc43fd9-b167-4f9c-ab90-11d1000a1e9f service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:45.811461 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5dc43fd9-b167-4f9c-ab90-11d1000a1e9f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:45.827273 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc43fd9-b167-4f9c-ab90-11d1000a1e9f service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:45.831231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:45.832047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:45.832333 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:45.832606 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] PATH_INFO: `/v3/projects/d4112b8acbd7461ab66bad4b72a044ad` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:45.833855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4112b8acbd7461ab66bad4b72a044ad)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:45.836575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:45.915348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2c414f4bff944e0180ce0c6aa4dffaf8', 'request_id': 'req-d3ef007d-7097-4702-8c08-8fd3e5babad4'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:45.920767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2c414f4bff944e0180ce0c6aa4dffaf8', 'request_id': 'req-d3ef007d-7097-4702-8c08-8fd3e5babad4'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:45.927482 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2c414f4bff944e0180ce0c6aa4dffaf8', 'request_id': 'req-d3ef007d-7097-4702-8c08-8fd3e5babad4'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:45.948394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4112b8acbd7461ab66bad4b72a044ad', 'request_id': 'req-011533ce-b7fa-4d14-be6e-07b0ba654c8b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:45.954526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4112b8acbd7461ab66bad4b72a044ad', 'request_id': 'req-011533ce-b7fa-4d14-be6e-07b0ba654c8b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.029795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] The token cache is being invalidate because project 2c414f4bff944e0180ce0c6aa4dffaf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:46.030121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.030803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.032576 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1334/2675] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:45 2022] DELETE /identity/v3/projects/2c414f4bff944e0180ce0c6aa4dffaf8 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:46.036127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:46.037614 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:46.075579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4112b8acbd7461ab66bad4b72a044ad', 'request_id': 'req-011533ce-b7fa-4d14-be6e-07b0ba654c8b'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.089265 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] The token cache is being invalidate because project d4112b8acbd7461ab66bad4b72a044ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:46.089826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.090821 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.092634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1342/2676] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:45 2022] DELETE /identity/v3/projects/d4112b8acbd7461ab66bad4b72a044ad => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:46.097047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:46.098637 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:46.107132 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:46.115042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ef007d-7097-4702-8c08-8fd3e5babad4 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:46.176029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011533ce-b7fa-4d14-be6e-07b0ba654c8b admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:46.177001 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:46.177806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:46.178225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:46.178812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] PATH_INFO: `/v3/users/9bfb2f4f49dd42ebaa5fe6d9067da27f` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:46.180511 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9bfb2f4f49dd42ebaa5fe6d9067da27f)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:46.182845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:46.182971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:46.183625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:46.183909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:46.184201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] PATH_INFO: `/v3/users/c1d9c9cca74c42149cb79439db49d3ab` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:46.185373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1d9c9cca74c42149cb79439db49d3ab)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:46.187360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:46.224293 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:46.369964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1d9c9cca74c42149cb79439db49d3ab'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.370678 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1d9c9cca74c42149cb79439db49d3ab'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.377836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1d9c9cca74c42149cb79439db49d3ab'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.378624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1d9c9cca74c42149cb79439db49d3ab'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.403097 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9bfb2f4f49dd42ebaa5fe6d9067da27f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.403752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9bfb2f4f49dd42ebaa5fe6d9067da27f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.412716 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9bfb2f4f49dd42ebaa5fe6d9067da27f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.426641 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9bfb2f4f49dd42ebaa5fe6d9067da27f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.452670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1d9c9cca74c42149cb79439db49d3ab'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.471240 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1343/2677] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:46 2022] DELETE /identity/v3/users/c1d9c9cca74c42149cb79439db49d3ab => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:46.556240 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9bfb2f4f49dd42ebaa5fe6d9067da27f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:46.571766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1335/2678] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:46 2022] DELETE /identity/v3/users/9bfb2f4f49dd42ebaa5fe6d9067da27f => generated 0 bytes in 532 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:47.037155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:47.040040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:47.171248 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3eeca63-2d4a-445f-9aba-dd6877bd2a9c admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:47.201143 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:47.231534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f98d6c-b916-49cb-a92a-ceb621dded94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:47.232203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f98d6c-b916-49cb-a92a-ceb621dded94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:47.232508 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f98d6c-b916-49cb-a92a-ceb621dded94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:47.232834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f98d6c-b916-49cb-a92a-ceb621dded94 admin admin] PATH_INFO: `/v3/projects/c3a499db865b45dba98acd7bb8a306d3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:47.234050 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f98d6c-b916-49cb-a92a-ceb621dded94 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c3a499db865b45dba98acd7bb8a306d3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:47.236010 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f98d6c-b916-49cb-a92a-ceb621dded94 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:47.261945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:47.264176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:47.283228 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f97ef1c-0b4f-4e2d-9915-9fe1aaaa151b admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:47.288526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:47.289456 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:47.289796 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:47.290124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] PATH_INFO: `/v3/projects/baeb5149de2f4caeb0f8ba621ea37885` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:47.291350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] RBAC: Authorizing `identity:delete_project(project_id=baeb5149de2f4caeb0f8ba621ea37885)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:47.293363 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:47.391930 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-e2f98d6c-b916-49cb-a92a-ceb621dded94 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c3a499db865b45dba98acd7bb8a306d3', 'request_id': 'req-e2f98d6c-b916-49cb-a92a-ceb621dded94'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.398543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-e2f98d6c-b916-49cb-a92a-ceb621dded94 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c3a499db865b45dba98acd7bb8a306d3', 'request_id': 'req-e2f98d6c-b916-49cb-a92a-ceb621dded94'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.423387 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'baeb5149de2f4caeb0f8ba621ea37885', 'request_id': 'req-bf34b2e0-0e66-478a-abdb-52eaad0faf38'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.429505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'baeb5149de2f4caeb0f8ba621ea37885', 'request_id': 'req-bf34b2e0-0e66-478a-abdb-52eaad0faf38'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.436546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'baeb5149de2f4caeb0f8ba621ea37885', 'request_id': 'req-bf34b2e0-0e66-478a-abdb-52eaad0faf38'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.538670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-e2f98d6c-b916-49cb-a92a-ceb621dded94 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c3a499db865b45dba98acd7bb8a306d3', 'request_id': 'req-e2f98d6c-b916-49cb-a92a-ceb621dded94'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.556720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-e2f98d6c-b916-49cb-a92a-ceb621dded94 admin admin] The token cache is being invalidate because project c3a499db865b45dba98acd7bb8a306d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:47.557110 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-e2f98d6c-b916-49cb-a92a-ceb621dded94 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.557839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-e2f98d6c-b916-49cb-a92a-ceb621dded94 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.559405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1344/2679] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:47 2022] DELETE /identity/v3/projects/c3a499db865b45dba98acd7bb8a306d3 => generated 0 bytes in 523 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:47.564457 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed7b883-5c87-4424-81e2-1c0101c82bd4 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:47.564848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed7b883-5c87-4424-81e2-1c0101c82bd4 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:47.565192 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed7b883-5c87-4424-81e2-1c0101c82bd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:47.585827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1ed7b883-5c87-4424-81e2-1c0101c82bd4 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:47.587184 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1ed7b883-5c87-4424-81e2-1c0101c82bd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:47.604876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:47.652272 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1345/2680] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:47.664577 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc07371a-65dc-4500-b6c5-e5812337036c None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:47.664784 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] The token cache is being invalidate because project baeb5149de2f4caeb0f8ba621ea37885 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:47.664921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc07371a-65dc-4500-b6c5-e5812337036c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:47.665120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc07371a-65dc-4500-b6c5-e5812337036c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:47.665346 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.666120 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.668036 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1336/2681] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:47 2022] DELETE /identity/v3/projects/baeb5149de2f4caeb0f8ba621ea37885 => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:47.671935 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:47.673768 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:47.687218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf34b2e0-0e66-478a-abdb-52eaad0faf38 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:47.693713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcbd221-b253-4b9f-a045-bf626b67e84d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:47.694704 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcbd221-b253-4b9f-a045-bf626b67e84d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:47.695199 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcbd221-b253-4b9f-a045-bf626b67e84d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:47.695797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcbd221-b253-4b9f-a045-bf626b67e84d admin admin] PATH_INFO: `/v3/users/8db64d4edc1c4b11a9346a4c5cec65e2` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:47.698009 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcbd221-b253-4b9f-a045-bf626b67e84d admin admin] RBAC: Authorizing `identity:delete_user(user_id=8db64d4edc1c4b11a9346a4c5cec65e2)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:47.699973 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcbd221-b253-4b9f-a045-bf626b67e84d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:47.701844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-bc07371a-65dc-4500-b6c5-e5812337036c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:47.703196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-bc07371a-65dc-4500-b6c5-e5812337036c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:47.712497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1346/2682] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:47.716695 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc07371a-65dc-4500-b6c5-e5812337036c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:47.726542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc07371a-65dc-4500-b6c5-e5812337036c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:47.731630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fa52a2-073e-43cd-a699-2b322af4819a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:47.732245 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fa52a2-073e-43cd-a699-2b322af4819a admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:47.732526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fa52a2-073e-43cd-a699-2b322af4819a admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:47.732796 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fa52a2-073e-43cd-a699-2b322af4819a admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:47.733439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fa52a2-073e-43cd-a699-2b322af4819a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:58:47.733741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fa52a2-073e-43cd-a699-2b322af4819a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:47.735407 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fa52a2-073e-43cd-a699-2b322af4819a admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:47.747860 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1347/2683] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:58:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:58:47.752171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fa52a2-073e-43cd-a699-2b322af4819a admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:47.761804 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fa52a2-073e-43cd-a699-2b322af4819a admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:47.766760 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18eb027-7b38-41e0-91c3-ea0e4875a882 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:47.767340 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18eb027-7b38-41e0-91c3-ea0e4875a882 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:47.767621 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18eb027-7b38-41e0-91c3-ea0e4875a882 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:47.767898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18eb027-7b38-41e0-91c3-ea0e4875a882 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:47.768515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18eb027-7b38-41e0-91c3-ea0e4875a882 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:47.770497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18eb027-7b38-41e0-91c3-ea0e4875a882 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:47.890576 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1348/2684] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:47 2022] POST /identity/v3/projects => generated 355 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:58:47.892714 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18eb027-7b38-41e0-91c3-ea0e4875a882 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:47.920447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0fcbd221-b253-4b9f-a045-bf626b67e84d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8db64d4edc1c4b11a9346a4c5cec65e2'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.921158 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0fcbd221-b253-4b9f-a045-bf626b67e84d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8db64d4edc1c4b11a9346a4c5cec65e2'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.929554 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0fcbd221-b253-4b9f-a045-bf626b67e84d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8db64d4edc1c4b11a9346a4c5cec65e2'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.942061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0fcbd221-b253-4b9f-a045-bf626b67e84d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8db64d4edc1c4b11a9346a4c5cec65e2'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:47.967933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18eb027-7b38-41e0-91c3-ea0e4875a882 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:47.972912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbabfc10-a27a-458a-afdd-3c82eb42c5fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:47.973630 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbabfc10-a27a-458a-afdd-3c82eb42c5fe admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:47.973978 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbabfc10-a27a-458a-afdd-3c82eb42c5fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:47.974289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbabfc10-a27a-458a-afdd-3c82eb42c5fe admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:47.974921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbabfc10-a27a-458a-afdd-3c82eb42c5fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:47.977058 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbabfc10-a27a-458a-afdd-3c82eb42c5fe admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:48.006288 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0fcbd221-b253-4b9f-a045-bf626b67e84d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8db64d4edc1c4b11a9346a4c5cec65e2'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:48.013212 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1337/2685] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:47 2022] DELETE /identity/v3/users/8db64d4edc1c4b11a9346a4c5cec65e2 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:48.014539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4ad93f-4f09-4c90-893e-7034b79185af None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:48.014826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4ad93f-4f09-4c90-893e-7034b79185af None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:48.015115 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4ad93f-4f09-4c90-893e-7034b79185af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:48.035575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-4d4ad93f-4f09-4c90-893e-7034b79185af None None] MFA Rules not processed for user `6683479f4e1c419e950fa7497f194ac1`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:48.036844 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4d4ad93f-4f09-4c90-893e-7034b79185af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:48.091676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1349/2686] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:47 2022] POST /identity/v3/users => generated 278 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:48.095805 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbabfc10-a27a-458a-afdd-3c82eb42c5fe admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:48.099890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-cbabfc10-a27a-458a-afdd-3c82eb42c5fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:48.111280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1338/2687] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:48 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:48.115479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbabfc10-a27a-458a-afdd-3c82eb42c5fe admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:48.201449 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e2ac62-c69b-40a6-82ce-437319058654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:48.201449 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e2ac62-c69b-40a6-82ce-437319058654 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:48.201874 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e2ac62-c69b-40a6-82ce-437319058654 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:48.201956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e2ac62-c69b-40a6-82ce-437319058654 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:48.202624 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e2ac62-c69b-40a6-82ce-437319058654 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:48.205097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e2ac62-c69b-40a6-82ce-437319058654 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:48.220571 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4ad93f-4f09-4c90-893e-7034b79185af None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:48.221114 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1350/2688] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:48 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:58:48.245391 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4ad93f-4f09-4c90-893e-7034b79185af None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:48.250469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57997acd-79be-42d2-a6d6-47a7d9b4a0c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:48.251189 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57997acd-79be-42d2-a6d6-47a7d9b4a0c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:48.251470 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57997acd-79be-42d2-a6d6-47a7d9b4a0c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:48.251744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57997acd-79be-42d2-a6d6-47a7d9b4a0c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:48.252566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57997acd-79be-42d2-a6d6-47a7d9b4a0c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:48.254687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57997acd-79be-42d2-a6d6-47a7d9b4a0c7 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:48.278093 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1339/2689] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:48 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:58:48.281768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e2ac62-c69b-40a6-82ce-437319058654 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:48.295138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e2ac62-c69b-40a6-82ce-437319058654 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:48.304848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3059a4-feca-4e2e-8ffa-952422d95174 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:48.306426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3059a4-feca-4e2e-8ffa-952422d95174 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:48.308328 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3059a4-feca-4e2e-8ffa-952422d95174 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:48.308760 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3059a4-feca-4e2e-8ffa-952422d95174 admin admin] PATH_INFO: `/v3/projects/adfea6e2a27e49559d531c8e0f131d2a/users/d769540a992b4eacb8e6a4aff286581c/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:48.345639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3059a4-feca-4e2e-8ffa-952422d95174 admin admin] RBAC: Authorizing `identity:create_grant(project_id=adfea6e2a27e49559d531c8e0f131d2a, user_id=d769540a992b4eacb8e6a4aff286581c, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:48.350709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3059a4-feca-4e2e-8ffa-952422d95174 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:48.428141 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1351/2690] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:48 2022] PUT /identity/v3/projects/adfea6e2a27e49559d531c8e0f131d2a/users/d769540a992b4eacb8e6a4aff286581c/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:48.439133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fddb60-c46b-44e7-b87f-7f2fc8688659 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:48.439429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fddb60-c46b-44e7-b87f-7f2fc8688659 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:48.439819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fddb60-c46b-44e7-b87f-7f2fc8688659 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:48.472404 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-91fddb60-c46b-44e7-b87f-7f2fc8688659 None None] MFA Rules not processed for user `d769540a992b4eacb8e6a4aff286581c`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:48.487267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-91fddb60-c46b-44e7-b87f-7f2fc8688659 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:48.578736 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3059a4-feca-4e2e-8ffa-952422d95174 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:48.580057 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1340/2691] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:48 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:48.647721 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3059a4-feca-4e2e-8ffa-952422d95174 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:48.653685 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4353a239-fc91-4345-bbf9-1e6fea3a1fcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:48.653954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4353a239-fc91-4345-bbf9-1e6fea3a1fcc admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:48.654289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4353a239-fc91-4345-bbf9-1e6fea3a1fcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:48.654566 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4353a239-fc91-4345-bbf9-1e6fea3a1fcc admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:48.655207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4353a239-fc91-4345-bbf9-1e6fea3a1fcc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:48.657270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4353a239-fc91-4345-bbf9-1e6fea3a1fcc admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:48.670376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fddb60-c46b-44e7-b87f-7f2fc8688659 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:48.672126 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-91fddb60-c46b-44e7-b87f-7f2fc8688659 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:48.694267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fddb60-c46b-44e7-b87f-7f2fc8688659 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:48.697489 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35e43da-9679-49f2-854e-a4753d8f9729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:48.698241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35e43da-9679-49f2-854e-a4753d8f9729 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:48.698451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35e43da-9679-49f2-854e-a4753d8f9729 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:48.698726 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35e43da-9679-49f2-854e-a4753d8f9729 admin admin] PATH_INFO: `/v3/projects/fa9c08c0d4a44eb19d78a4c3c2bb5dd7` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:48.699919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35e43da-9679-49f2-854e-a4753d8f9729 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa9c08c0d4a44eb19d78a4c3c2bb5dd7)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:48.701838 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35e43da-9679-49f2-854e-a4753d8f9729 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:48.753533 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1352/2692] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:48 2022] POST /identity/v3/projects => generated 357 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:48.758299 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4353a239-fc91-4345-bbf9-1e6fea3a1fcc admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:48.827227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4353a239-fc91-4345-bbf9-1e6fea3a1fcc admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:48.832071 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d484c658-ba98-4501-8eb6-671f37d4d0df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:48.832669 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d484c658-ba98-4501-8eb6-671f37d4d0df admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:48.832945 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d484c658-ba98-4501-8eb6-671f37d4d0df admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:48.833298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d484c658-ba98-4501-8eb6-671f37d4d0df admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:48.833824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d484c658-ba98-4501-8eb6-671f37d4d0df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:48.836847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d484c658-ba98-4501-8eb6-671f37d4d0df admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:48.862046 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a35e43da-9679-49f2-854e-a4753d8f9729 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa9c08c0d4a44eb19d78a4c3c2bb5dd7', 'request_id': 'req-a35e43da-9679-49f2-854e-a4753d8f9729'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:48.868065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a35e43da-9679-49f2-854e-a4753d8f9729 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa9c08c0d4a44eb19d78a4c3c2bb5dd7', 'request_id': 'req-a35e43da-9679-49f2-854e-a4753d8f9729'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:48.874509 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a35e43da-9679-49f2-854e-a4753d8f9729 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa9c08c0d4a44eb19d78a4c3c2bb5dd7', 'request_id': 'req-a35e43da-9679-49f2-854e-a4753d8f9729'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:48.904508 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1353/2693] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:48 2022] POST /identity/v3/users => generated 285 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:58:48.908671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d484c658-ba98-4501-8eb6-671f37d4d0df admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:48.926120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d484c658-ba98-4501-8eb6-671f37d4d0df admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:48.944872 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a35e43da-9679-49f2-854e-a4753d8f9729 admin admin] The token cache is being invalidate because project fa9c08c0d4a44eb19d78a4c3c2bb5dd7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:48.945724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a35e43da-9679-49f2-854e-a4753d8f9729 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:48.946392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-a35e43da-9679-49f2-854e-a4753d8f9729 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:48.948615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1341/2694] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:48 2022] DELETE /identity/v3/projects/fa9c08c0d4a44eb19d78a4c3c2bb5dd7 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:48.954889 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aee63c1-e099-4261-b4a2-45e07716aa00 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:48.955205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aee63c1-e099-4261-b4a2-45e07716aa00 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:48.955860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aee63c1-e099-4261-b4a2-45e07716aa00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:48.972462 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:48.978882 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-9aee63c1-e099-4261-b4a2-45e07716aa00 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:48.980099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-9aee63c1-e099-4261-b4a2-45e07716aa00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:49.001064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1722b-0451-4077-b64f-96ccc58536a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.001672 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1722b-0451-4077-b64f-96ccc58536a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.001948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1722b-0451-4077-b64f-96ccc58536a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.002255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a1722b-0451-4077-b64f-96ccc58536a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.002896 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a1722b-0451-4077-b64f-96ccc58536a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.004493 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a1722b-0451-4077-b64f-96ccc58536a3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.015120 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1354/2695] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:48 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:58:49.018952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1722b-0451-4077-b64f-96ccc58536a3 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:49.020772 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-42a1722b-0451-4077-b64f-96ccc58536a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:49.033676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a1722b-0451-4077-b64f-96ccc58536a3 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:49.039578 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db598ff9-cb46-4e43-945e-1d070cf1cfba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.040250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db598ff9-cb46-4e43-945e-1d070cf1cfba admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.040526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db598ff9-cb46-4e43-945e-1d070cf1cfba admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.042767 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db598ff9-cb46-4e43-945e-1d070cf1cfba admin admin] PATH_INFO: `/v3/projects/7b363b6ea1ae4859b3d682deca8bbd9c/users/5114f45f4ae14962a38d5bfe85e5dde7/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.045862 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1342/2696] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:49.054960 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975ab202-389a-451b-a867-6bc2e5b3f5ba None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.055304 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975ab202-389a-451b-a867-6bc2e5b3f5ba None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.055589 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975ab202-389a-451b-a867-6bc2e5b3f5ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.058751 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db598ff9-cb46-4e43-945e-1d070cf1cfba admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b363b6ea1ae4859b3d682deca8bbd9c, user_id=5114f45f4ae14962a38d5bfe85e5dde7, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.061500 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db598ff9-cb46-4e43-945e-1d070cf1cfba admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.075215 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-975ab202-389a-451b-a867-6bc2e5b3f5ba None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:49.076441 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-975ab202-389a-451b-a867-6bc2e5b3f5ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:49.086219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1343/2697] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:49.090623 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975ab202-389a-451b-a867-6bc2e5b3f5ba None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:49.101238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975ab202-389a-451b-a867-6bc2e5b3f5ba None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:49.106410 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d0807a-8ff1-49f6-8377-ddb5fc3bff68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.107112 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d0807a-8ff1-49f6-8377-ddb5fc3bff68 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.107449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d0807a-8ff1-49f6-8377-ddb5fc3bff68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.107822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d0807a-8ff1-49f6-8377-ddb5fc3bff68 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.108545 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d0807a-8ff1-49f6-8377-ddb5fc3bff68 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:58:49.108910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d0807a-8ff1-49f6-8377-ddb5fc3bff68 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.110692 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d0807a-8ff1-49f6-8377-ddb5fc3bff68 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.134668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1344/2698] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:58:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 03:58:49.143749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d0807a-8ff1-49f6-8377-ddb5fc3bff68 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:49.157249 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d0807a-8ff1-49f6-8377-ddb5fc3bff68 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:49.168356 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1355/2699] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:49 2022] PUT /identity/v3/projects/7b363b6ea1ae4859b3d682deca8bbd9c/users/5114f45f4ae14962a38d5bfe85e5dde7/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:49.169486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db598ff9-cb46-4e43-945e-1d070cf1cfba admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:49.185438 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=77,keystone:DELETE=20,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:49.240395 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1faf7d-5859-46ef-af34-95e91a5c713e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.241100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1faf7d-5859-46ef-af34-95e91a5c713e admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.241373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1faf7d-5859-46ef-af34-95e91a5c713e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.241644 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1faf7d-5859-46ef-af34-95e91a5c713e admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.242303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1faf7d-5859-46ef-af34-95e91a5c713e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.244188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1faf7d-5859-46ef-af34-95e91a5c713e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.247292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db598ff9-cb46-4e43-945e-1d070cf1cfba admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:49.253706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8b374c-9f3f-46b5-b7fc-7489681b7307 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.254358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8b374c-9f3f-46b5-b7fc-7489681b7307 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.254656 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8b374c-9f3f-46b5-b7fc-7489681b7307 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.254921 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8b374c-9f3f-46b5-b7fc-7489681b7307 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.255573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8b374c-9f3f-46b5-b7fc-7489681b7307 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.257653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8b374c-9f3f-46b5-b7fc-7489681b7307 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.269164 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1356/2700] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:49 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:49.274190 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8b374c-9f3f-46b5-b7fc-7489681b7307 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:49.283826 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8b374c-9f3f-46b5-b7fc-7489681b7307 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:49.289953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dfacb3-bc91-42de-a9df-caaf5a705ed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.290664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dfacb3-bc91-42de-a9df-caaf5a705ed4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.290935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dfacb3-bc91-42de-a9df-caaf5a705ed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.291205 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dfacb3-bc91-42de-a9df-caaf5a705ed4 admin admin] PATH_INFO: `/v3/projects/7b363b6ea1ae4859b3d682deca8bbd9c/users/5114f45f4ae14962a38d5bfe85e5dde7/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.293170 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19dfacb3-bc91-42de-a9df-caaf5a705ed4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b363b6ea1ae4859b3d682deca8bbd9c, user_id=5114f45f4ae14962a38d5bfe85e5dde7, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.295872 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19dfacb3-bc91-42de-a9df-caaf5a705ed4 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.306634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1357/2701] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:49 2022] PUT /identity/v3/projects/7b363b6ea1ae4859b3d682deca8bbd9c/users/5114f45f4ae14962a38d5bfe85e5dde7/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:58:49.313808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1345/2702] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:49 2022] POST /identity/v3/projects => generated 401 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:49.314785 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dfacb3-bc91-42de-a9df-caaf5a705ed4 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:49.317707 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400fb28e-2a71-46b3-9b2a-f6c71f0d15b4 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.317995 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400fb28e-2a71-46b3-9b2a-f6c71f0d15b4 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.318327 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400fb28e-2a71-46b3-9b2a-f6c71f0d15b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.346730 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-400fb28e-2a71-46b3-9b2a-f6c71f0d15b4 None None] MFA Rules not processed for user `5114f45f4ae14962a38d5bfe85e5dde7`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:49.348101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-400fb28e-2a71-46b3-9b2a-f6c71f0d15b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:49.383234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dfacb3-bc91-42de-a9df-caaf5a705ed4 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:49.419577 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1346/2703] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:49 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:49.432808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400fb28e-2a71-46b3-9b2a-f6c71f0d15b4 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:49.434239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-400fb28e-2a71-46b3-9b2a-f6c71f0d15b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:49.447067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da26096e-9cd1-40cc-a98b-da797e2ad958 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.447729 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da26096e-9cd1-40cc-a98b-da797e2ad958 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.448014 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da26096e-9cd1-40cc-a98b-da797e2ad958 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.448287 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da26096e-9cd1-40cc-a98b-da797e2ad958 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.448919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da26096e-9cd1-40cc-a98b-da797e2ad958 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.450923 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da26096e-9cd1-40cc-a98b-da797e2ad958 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.494530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400fb28e-2a71-46b3-9b2a-f6c71f0d15b4 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:49.498366 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa32644-dc61-41f6-84ab-5d7d60cf4c31 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.499002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa32644-dc61-41f6-84ab-5d7d60cf4c31 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.499280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa32644-dc61-41f6-84ab-5d7d60cf4c31 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.499552 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa32644-dc61-41f6-84ab-5d7d60cf4c31 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.509781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa32644-dc61-41f6-84ab-5d7d60cf4c31 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.511275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa32644-dc61-41f6-84ab-5d7d60cf4c31 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.522451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1347/2704] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:49 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:49.530915 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1358/2705] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:49 2022] POST /identity/v3/users => generated 301 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:58:49.534721 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa32644-dc61-41f6-84ab-5d7d60cf4c31 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:49.545745 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa32644-dc61-41f6-84ab-5d7d60cf4c31 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:49.550574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d61853-f4c0-46b2-a0c0-c668d14b269e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.551184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d61853-f4c0-46b2-a0c0-c668d14b269e admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.551478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d61853-f4c0-46b2-a0c0-c668d14b269e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.551869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d61853-f4c0-46b2-a0c0-c668d14b269e admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.552521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d61853-f4c0-46b2-a0c0-c668d14b269e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.555241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d61853-f4c0-46b2-a0c0-c668d14b269e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.565763 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1348/2706] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:49 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:49.570277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da26096e-9cd1-40cc-a98b-da797e2ad958 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:49.580531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da26096e-9cd1-40cc-a98b-da797e2ad958 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:49.586259 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b7da9b-4d98-43ac-b324-1693f5fa61b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.588004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b7da9b-4d98-43ac-b324-1693f5fa61b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.588004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b7da9b-4d98-43ac-b324-1693f5fa61b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.588004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b7da9b-4d98-43ac-b324-1693f5fa61b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.588004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b7da9b-4d98-43ac-b324-1693f5fa61b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.589588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b7da9b-4d98-43ac-b324-1693f5fa61b1 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.603934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1359/2707] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:49.605740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d61853-f4c0-46b2-a0c0-c668d14b269e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:49.618198 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d61853-f4c0-46b2-a0c0-c668d14b269e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:49.623277 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc24aa5-5bec-4631-b187-3472c7ccec7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.624080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc24aa5-5bec-4631-b187-3472c7ccec7d admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.624369 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc24aa5-5bec-4631-b187-3472c7ccec7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.624892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc24aa5-5bec-4631-b187-3472c7ccec7d admin admin] PATH_INFO: `/v3/projects/f0969aac6eac4b299f2df631565c71a7/users/e24ec85bd0884010924789414a13930d/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.648786 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc24aa5-5bec-4631-b187-3472c7ccec7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0969aac6eac4b299f2df631565c71a7, user_id=e24ec85bd0884010924789414a13930d, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.653218 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc24aa5-5bec-4631-b187-3472c7ccec7d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.730916 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b7da9b-4d98-43ac-b324-1693f5fa61b1 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:49.738923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1349/2708] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:49 2022] PUT /identity/v3/projects/f0969aac6eac4b299f2df631565c71a7/users/e24ec85bd0884010924789414a13930d/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:49.746843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b7da9b-4d98-43ac-b324-1693f5fa61b1 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:49.751954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c41810c-18e7-49fe-aeae-33f02fbbbdd8 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.752317 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c41810c-18e7-49fe-aeae-33f02fbbbdd8 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.752586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c41810c-18e7-49fe-aeae-33f02fbbbdd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.783229 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-6c41810c-18e7-49fe-aeae-33f02fbbbdd8 None None] MFA Rules not processed for user `e24ec85bd0884010924789414a13930d`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:49.784536 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6c41810c-18e7-49fe-aeae-33f02fbbbdd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:49.802629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:49.808309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ccb910-43fb-44f5-a1cf-da8f1b0da596 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.808954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ccb910-43fb-44f5-a1cf-da8f1b0da596 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.809235 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ccb910-43fb-44f5-a1cf-da8f1b0da596 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.809565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ccb910-43fb-44f5-a1cf-da8f1b0da596 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.811426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-56ccb910-43fb-44f5-a1cf-da8f1b0da596 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:49.860820 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:49.870672 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1350/2709] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:49 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:49.875131 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c41810c-18e7-49fe-aeae-33f02fbbbdd8 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:49.908035 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ccb910-43fb-44f5-a1cf-da8f1b0da596 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.909474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ccb910-43fb-44f5-a1cf-da8f1b0da596 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.920128 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1360/2710] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:49 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:58:49.947814 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c41810c-18e7-49fe-aeae-33f02fbbbdd8 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:49.952753 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28167033-df4f-482a-af89-0c1fa5b27ad8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:49.953380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28167033-df4f-482a-af89-0c1fa5b27ad8 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:49.953662 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28167033-df4f-482a-af89-0c1fa5b27ad8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:49.953933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28167033-df4f-482a-af89-0c1fa5b27ad8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:49.954595 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28167033-df4f-482a-af89-0c1fa5b27ad8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:49.956477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28167033-df4f-482a-af89-0c1fa5b27ad8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:49.961747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:50.077209 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1351/2711] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:49 2022] POST /identity/v3/projects => generated 399 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:58:50.081663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ccb910-43fb-44f5-a1cf-da8f1b0da596 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:50.093890 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ccb910-43fb-44f5-a1cf-da8f1b0da596 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:50.156429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40f204f-9c89-4eb2-89d8-c32562c34768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:50.157189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40f204f-9c89-4eb2-89d8-c32562c34768 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:50.157769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40f204f-9c89-4eb2-89d8-c32562c34768 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:50.158268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40f204f-9c89-4eb2-89d8-c32562c34768 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:50.158904 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40f204f-9c89-4eb2-89d8-c32562c34768 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:50.160819 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40f204f-9c89-4eb2-89d8-c32562c34768 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:50.207266 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:50.253393 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1361/2712] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:50 2022] POST /identity/v3/users => generated 306 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:50.265280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28167033-df4f-482a-af89-0c1fa5b27ad8 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:50.276530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28167033-df4f-482a-af89-0c1fa5b27ad8 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:50.282574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae45b6e-ce64-4543-885e-6217a8c87886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:50.283186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae45b6e-ce64-4543-885e-6217a8c87886 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:50.283466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae45b6e-ce64-4543-885e-6217a8c87886 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:50.283731 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae45b6e-ce64-4543-885e-6217a8c87886 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:50.285379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae45b6e-ce64-4543-885e-6217a8c87886 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:50.288534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae45b6e-ce64-4543-885e-6217a8c87886 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:50.309699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1352/2713] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:50 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:50.310377 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:50.312089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40f204f-9c89-4eb2-89d8-c32562c34768 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:50.331497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40f204f-9c89-4eb2-89d8-c32562c34768 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:50.331497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cee925f-e82f-4355-8e8b-8636786164c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:50.332104 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cee925f-e82f-4355-8e8b-8636786164c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:50.332388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cee925f-e82f-4355-8e8b-8636786164c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:50.332664 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cee925f-e82f-4355-8e8b-8636786164c5 admin admin] PATH_INFO: `/v3/projects/ab070ff60e804f709154a5e9fd8fadb0/users/c63592ac5e2e422898585031677f1f5a/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:50.353021 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cee925f-e82f-4355-8e8b-8636786164c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab070ff60e804f709154a5e9fd8fadb0, user_id=c63592ac5e2e422898585031677f1f5a, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:50.356266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cee925f-e82f-4355-8e8b-8636786164c5 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:50.429891 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1362/2714] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:50 2022] PUT /identity/v3/projects/ab070ff60e804f709154a5e9fd8fadb0/users/c63592ac5e2e422898585031677f1f5a/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:50.433921 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae45b6e-ce64-4543-885e-6217a8c87886 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:50.445192 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae45b6e-ce64-4543-885e-6217a8c87886 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:50.508872 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced8d04d-43bc-4b0e-bfb0-f212a65fbed1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:50.509465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced8d04d-43bc-4b0e-bfb0-f212a65fbed1 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:50.509733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced8d04d-43bc-4b0e-bfb0-f212a65fbed1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:50.509997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced8d04d-43bc-4b0e-bfb0-f212a65fbed1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:50.510658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced8d04d-43bc-4b0e-bfb0-f212a65fbed1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:50.512220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced8d04d-43bc-4b0e-bfb0-f212a65fbed1 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:50.521902 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1353/2715] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:50 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:50.525650 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:50.526543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cee925f-e82f-4355-8e8b-8636786164c5 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:50.537464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cee925f-e82f-4355-8e8b-8636786164c5 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:50.544724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb67c7e6-be27-4607-92c8-dd73bccfab26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:50.545395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb67c7e6-be27-4607-92c8-dd73bccfab26 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:50.545668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb67c7e6-be27-4607-92c8-dd73bccfab26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:50.545973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb67c7e6-be27-4607-92c8-dd73bccfab26 admin admin] PATH_INFO: `/v3/projects/ab070ff60e804f709154a5e9fd8fadb0/users/c63592ac5e2e422898585031677f1f5a/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:50.547898 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb67c7e6-be27-4607-92c8-dd73bccfab26 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab070ff60e804f709154a5e9fd8fadb0, user_id=c63592ac5e2e422898585031677f1f5a, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:50.550769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb67c7e6-be27-4607-92c8-dd73bccfab26 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:50.563303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1363/2716] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:50 2022] PUT /identity/v3/projects/ab070ff60e804f709154a5e9fd8fadb0/users/c63592ac5e2e422898585031677f1f5a/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:50.574768 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e99ce1-8db8-4312-b9f2-0aaab325b670 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:50.575055 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e99ce1-8db8-4312-b9f2-0aaab325b670 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:50.575331 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e99ce1-8db8-4312-b9f2-0aaab325b670 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:50.605501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-01e99ce1-8db8-4312-b9f2-0aaab325b670 None None] MFA Rules not processed for user `c63592ac5e2e422898585031677f1f5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:50.606872 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-01e99ce1-8db8-4312-b9f2-0aaab325b670 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:50.680845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1354/2717] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:50 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:50.692183 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb67c7e6-be27-4607-92c8-dd73bccfab26 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:50.779206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb67c7e6-be27-4607-92c8-dd73bccfab26 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:50.783350 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5062fac5-5da3-4fe8-b831-dd0792136dc5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:50.783968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5062fac5-5da3-4fe8-b831-dd0792136dc5 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:50.784255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5062fac5-5da3-4fe8-b831-dd0792136dc5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:50.784526 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5062fac5-5da3-4fe8-b831-dd0792136dc5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:50.796512 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5062fac5-5da3-4fe8-b831-dd0792136dc5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:50.798219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5062fac5-5da3-4fe8-b831-dd0792136dc5 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:50.809975 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1364/2718] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:50 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:51.001696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e99ce1-8db8-4312-b9f2-0aaab325b670 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:51.012400 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e99ce1-8db8-4312-b9f2-0aaab325b670 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:51.015910 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a37c870-8449-44b3-9fdc-72f5884ccb10 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:51.016501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a37c870-8449-44b3-9fdc-72f5884ccb10 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:51.016777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a37c870-8449-44b3-9fdc-72f5884ccb10 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:51.017048 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a37c870-8449-44b3-9fdc-72f5884ccb10 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:51.074330 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a37c870-8449-44b3-9fdc-72f5884ccb10 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:51.075751 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a37c870-8449-44b3-9fdc-72f5884ccb10 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:51.084880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1355/2719] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:51 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:54.860947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5062fac5-5da3-4fe8-b831-dd0792136dc5 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:54.862610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5062fac5-5da3-4fe8-b831-dd0792136dc5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:54.897750 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:54.926984 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5062fac5-5da3-4fe8-b831-dd0792136dc5 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:54.931865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991f8a10-44fe-4b28-8d83-4386ed31d63a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:54.932556 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991f8a10-44fe-4b28-8d83-4386ed31d63a service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:54.933018 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991f8a10-44fe-4b28-8d83-4386ed31d63a service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:54.933301 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991f8a10-44fe-4b28-8d83-4386ed31d63a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:54.935649 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-991f8a10-44fe-4b28-8d83-4386ed31d63a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:54.946957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991f8a10-44fe-4b28-8d83-4386ed31d63a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:54.948505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991f8a10-44fe-4b28-8d83-4386ed31d63a service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:54.957458 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1365/2720] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:54 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:56.343561 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a37c870-8449-44b3-9fdc-72f5884ccb10 service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:56.352771 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:56.407013 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a37c870-8449-44b3-9fdc-72f5884ccb10 service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:56.413687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:56.413687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:56.414002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:56.414247 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] PATH_INFO: `/v3/users/e24ec85bd0884010924789414a13930d` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:56.415640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=e24ec85bd0884010924789414a13930d)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:56.417572 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:56.621827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e24ec85bd0884010924789414a13930d'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:56.622553 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e24ec85bd0884010924789414a13930d'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:56.629625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e24ec85bd0884010924789414a13930d'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:56.657281 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e24ec85bd0884010924789414a13930d'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:56.731402 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e24ec85bd0884010924789414a13930d'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:56.738243 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1356/2721] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:56 2022] DELETE /identity/v3/users/e24ec85bd0884010924789414a13930d => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:56.743720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f03e267-4078-4be0-b53f-88a321e4a58b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:56.744289 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f03e267-4078-4be0-b53f-88a321e4a58b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:56.744479 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f03e267-4078-4be0-b53f-88a321e4a58b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:56.766138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-3f03e267-4078-4be0-b53f-88a321e4a58b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:56.769222 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3f03e267-4078-4be0-b53f-88a321e4a58b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:56.831748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1366/2722] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:58:56.841671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:56.843124 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:56.904264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7b4e35-7834-49fd-be3c-342cb1dde6aa admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:56.907955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe268859-6547-4de0-b648-73252503de8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:56.908558 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe268859-6547-4de0-b648-73252503de8d service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:56.908834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe268859-6547-4de0-b648-73252503de8d service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:56.909102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe268859-6547-4de0-b648-73252503de8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:56.919248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe268859-6547-4de0-b648-73252503de8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:56.920705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe268859-6547-4de0-b648-73252503de8d service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:56.931856 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1357/2723] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:57.341043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f03e267-4078-4be0-b53f-88a321e4a58b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:57.342544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3f03e267-4078-4be0-b53f-88a321e4a58b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:57.357001 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f03e267-4078-4be0-b53f-88a321e4a58b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:57.362836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:57.363551 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:57.363788 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:57.364140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] PATH_INFO: `/v3/projects/f0969aac6eac4b299f2df631565c71a7` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:57.370671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0969aac6eac4b299f2df631565c71a7)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:57.370671 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:57.499603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0969aac6eac4b299f2df631565c71a7', 'request_id': 'req-46b207e9-8284-41b5-8d4e-bde294da593e'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:57.505085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0969aac6eac4b299f2df631565c71a7', 'request_id': 'req-46b207e9-8284-41b5-8d4e-bde294da593e'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:57.570957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0969aac6eac4b299f2df631565c71a7', 'request_id': 'req-46b207e9-8284-41b5-8d4e-bde294da593e'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:57.583302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] The token cache is being invalidate because project f0969aac6eac4b299f2df631565c71a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:57.583592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:57.584402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:57.586716 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1367/2724] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:57 2022] DELETE /identity/v3/projects/f0969aac6eac4b299f2df631565c71a7 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:57.590894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe268859-6547-4de0-b648-73252503de8d service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:57.600846 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe268859-6547-4de0-b648-73252503de8d service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:57.628568 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:57.662490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:57.669117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:57.669117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:57.669117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] PATH_INFO: `/v3/users/c63592ac5e2e422898585031677f1f5a` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:57.669117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c63592ac5e2e422898585031677f1f5a)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:57.669117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:57.947326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c63592ac5e2e422898585031677f1f5a'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:57.948035 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c63592ac5e2e422898585031677f1f5a'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:57.956062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c63592ac5e2e422898585031677f1f5a'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:57.972475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c63592ac5e2e422898585031677f1f5a'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:58.063178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c63592ac5e2e422898585031677f1f5a'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:58.069929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1358/2725] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:58:57 2022] DELETE /identity/v3/users/c63592ac5e2e422898585031677f1f5a => generated 0 bytes in 480 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:58.260089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:58.261588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:58.321261 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b207e9-8284-41b5-8d4e-bde294da593e admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:58.326102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:58.376504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066337f0-c25b-4f9c-be29-079be22afca3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:58.377034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066337f0-c25b-4f9c-be29-079be22afca3 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:58.377313 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066337f0-c25b-4f9c-be29-079be22afca3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:58.377631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066337f0-c25b-4f9c-be29-079be22afca3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:58.379481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-066337f0-c25b-4f9c-be29-079be22afca3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:58.454956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066337f0-c25b-4f9c-be29-079be22afca3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:58.456415 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066337f0-c25b-4f9c-be29-079be22afca3 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:58.467171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1368/2726] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:58:58 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:58:58.613873 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:58.615615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:58.739797 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c886c8a2-284e-4775-80ba-720fa5a4e121 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:58.752148 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87db75b0-5934-471b-8f08-a98619ee722c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:58.752812 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87db75b0-5934-471b-8f08-a98619ee722c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:58.753097 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87db75b0-5934-471b-8f08-a98619ee722c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:58.753400 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87db75b0-5934-471b-8f08-a98619ee722c admin admin] PATH_INFO: `/v3/projects/ab070ff60e804f709154a5e9fd8fadb0` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:58.756398 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87db75b0-5934-471b-8f08-a98619ee722c admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab070ff60e804f709154a5e9fd8fadb0)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:58.762490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87db75b0-5934-471b-8f08-a98619ee722c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:58.904893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-87db75b0-5934-471b-8f08-a98619ee722c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab070ff60e804f709154a5e9fd8fadb0', 'request_id': 'req-87db75b0-5934-471b-8f08-a98619ee722c'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:58.906111 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:58.911486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-87db75b0-5934-471b-8f08-a98619ee722c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab070ff60e804f709154a5e9fd8fadb0', 'request_id': 'req-87db75b0-5934-471b-8f08-a98619ee722c'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:58.919455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-87db75b0-5934-471b-8f08-a98619ee722c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab070ff60e804f709154a5e9fd8fadb0', 'request_id': 'req-87db75b0-5934-471b-8f08-a98619ee722c'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:59.034356 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-87db75b0-5934-471b-8f08-a98619ee722c admin admin] The token cache is being invalidate because project ab070ff60e804f709154a5e9fd8fadb0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:58:59.034646 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-87db75b0-5934-471b-8f08-a98619ee722c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:59.035367 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-87db75b0-5934-471b-8f08-a98619ee722c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:58:59.037368 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1359/2727] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:58:58 2022] DELETE /identity/v3/projects/ab070ff60e804f709154a5e9fd8fadb0 => generated 0 bytes in 424 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:59.042041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc0736d-7353-46df-9130-ef4768c4089a None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:59.042364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc0736d-7353-46df-9130-ef4768c4089a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:59.042641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc0736d-7353-46df-9130-ef4768c4089a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:59.063537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-cfc0736d-7353-46df-9130-ef4768c4089a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:59.066019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-cfc0736d-7353-46df-9130-ef4768c4089a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:59.132096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1369/2728] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:59.140453 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4000114-87b2-4fc9-b953-d6f33010d757 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:59.140809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4000114-87b2-4fc9-b953-d6f33010d757 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:59.141093 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4000114-87b2-4fc9-b953-d6f33010d757 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:59.163307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f4000114-87b2-4fc9-b953-d6f33010d757 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:59.164847 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f4000114-87b2-4fc9-b953-d6f33010d757 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:59.173558 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1360/2729] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:58:59.177986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc0736d-7353-46df-9130-ef4768c4089a None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:59.188574 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc0736d-7353-46df-9130-ef4768c4089a None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:59.193477 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a654d58e-d337-41c4-8f5b-11e265c40a1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:59.195668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a654d58e-d337-41c4-8f5b-11e265c40a1f admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:59.195668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a654d58e-d337-41c4-8f5b-11e265c40a1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:59.195668 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a654d58e-d337-41c4-8f5b-11e265c40a1f admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:59.195949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a654d58e-d337-41c4-8f5b-11e265c40a1f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:58:59.196260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a654d58e-d337-41c4-8f5b-11e265c40a1f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:59.197988 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a654d58e-d337-41c4-8f5b-11e265c40a1f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:59.217666 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1370/2730] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:58:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:58:59.219780 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4000114-87b2-4fc9-b953-d6f33010d757 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:59.239478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4000114-87b2-4fc9-b953-d6f33010d757 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:59.243025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82de6ba-6443-47c4-af75-622414963348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:59.243866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82de6ba-6443-47c4-af75-622414963348 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:59.244149 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82de6ba-6443-47c4-af75-622414963348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:59.244449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82de6ba-6443-47c4-af75-622414963348 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:59.245170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82de6ba-6443-47c4-af75-622414963348 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:59.247114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82de6ba-6443-47c4-af75-622414963348 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:59.351214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1361/2731] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:58:59 2022] POST /identity/v3/projects => generated 359 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:58:59.356091 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a654d58e-d337-41c4-8f5b-11e265c40a1f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:59.366832 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a654d58e-d337-41c4-8f5b-11e265c40a1f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:59.424193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=100 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:58:59.436570 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b6f963-ba96-4301-8469-cce29a7aa532 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:59.439015 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b6f963-ba96-4301-8469-cce29a7aa532 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:59.439015 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b6f963-ba96-4301-8469-cce29a7aa532 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:59.439015 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b6f963-ba96-4301-8469-cce29a7aa532 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:59.439015 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b6f963-ba96-4301-8469-cce29a7aa532 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:59.441097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b6f963-ba96-4301-8469-cce29a7aa532 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:59.515504 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1371/2732] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:58:59 2022] POST /identity/v3/users => generated 280 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:58:59.519807 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82de6ba-6443-47c4-af75-622414963348 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:59.531542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82de6ba-6443-47c4-af75-622414963348 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:59.541949 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9740b4ba-f40a-47be-8320-3e42e1bfeb73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:59.543509 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9740b4ba-f40a-47be-8320-3e42e1bfeb73 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:59.544888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9740b4ba-f40a-47be-8320-3e42e1bfeb73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:59.545275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9740b4ba-f40a-47be-8320-3e42e1bfeb73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:59.547526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9740b4ba-f40a-47be-8320-3e42e1bfeb73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:59.549053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9740b4ba-f40a-47be-8320-3e42e1bfeb73 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:59.562966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1362/2733] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:59 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:58:59.568382 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b6f963-ba96-4301-8469-cce29a7aa532 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:59.579167 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b6f963-ba96-4301-8469-cce29a7aa532 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:59.589942 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7445e50f-8a94-49f6-a285-f9fc07b50ce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:59.590603 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7445e50f-8a94-49f6-a285-f9fc07b50ce2 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:59.590917 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7445e50f-8a94-49f6-a285-f9fc07b50ce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:59.591193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7445e50f-8a94-49f6-a285-f9fc07b50ce2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:59.591848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7445e50f-8a94-49f6-a285-f9fc07b50ce2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:59.593538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7445e50f-8a94-49f6-a285-f9fc07b50ce2 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:59.605828 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1372/2734] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:58:59 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:58:59.610033 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9740b4ba-f40a-47be-8320-3e42e1bfeb73 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:58:59.620789 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9740b4ba-f40a-47be-8320-3e42e1bfeb73 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:58:59.627060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7eabdf-ca4b-4d22-9880-7bdbd6f8b8bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:58:59.627702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7eabdf-ca4b-4d22-9880-7bdbd6f8b8bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:59.628076 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7eabdf-ca4b-4d22-9880-7bdbd6f8b8bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:59.628811 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7eabdf-ca4b-4d22-9880-7bdbd6f8b8bc admin admin] PATH_INFO: `/v3/projects/2be393c7a74a45828c2086f843aa1486/users/83f3e29a1f304013ac3b451b8fcd54fb/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:59.644677 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7eabdf-ca4b-4d22-9880-7bdbd6f8b8bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=2be393c7a74a45828c2086f843aa1486, user_id=83f3e29a1f304013ac3b451b8fcd54fb, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:58:59.647444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7eabdf-ca4b-4d22-9880-7bdbd6f8b8bc admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:58:59.797102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1363/2735] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:58:59 2022] PUT /identity/v3/projects/2be393c7a74a45828c2086f843aa1486/users/83f3e29a1f304013ac3b451b8fcd54fb/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:58:59.865089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e04ef47-814c-4cba-b169-71a38c70fa0c None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:58:59.865381 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e04ef47-814c-4cba-b169-71a38c70fa0c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:58:59.865646 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e04ef47-814c-4cba-b169-71a38c70fa0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:58:59.897059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1e04ef47-814c-4cba-b169-71a38c70fa0c None None] MFA Rules not processed for user `83f3e29a1f304013ac3b451b8fcd54fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:58:59.898369 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1e04ef47-814c-4cba-b169-71a38c70fa0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:58:59.969172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1373/2736] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:58:59 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:00.002202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7eabdf-ca4b-4d22-9880-7bdbd6f8b8bc admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:00.004231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-3c7eabdf-ca4b-4d22-9880-7bdbd6f8b8bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:00.068345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7eabdf-ca4b-4d22-9880-7bdbd6f8b8bc admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:00.072290 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0719fd-b42f-437c-a929-7e1f2504edf6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:00.072899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0719fd-b42f-437c-a929-7e1f2504edf6 service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:00.073173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0719fd-b42f-437c-a929-7e1f2504edf6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:00.073441 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0719fd-b42f-437c-a929-7e1f2504edf6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:00.082283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0719fd-b42f-437c-a929-7e1f2504edf6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:00.083811 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0719fd-b42f-437c-a929-7e1f2504edf6 service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:00.093465 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1364/2737] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:00 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:00.110326 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e04ef47-814c-4cba-b169-71a38c70fa0c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:00.182281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e04ef47-814c-4cba-b169-71a38c70fa0c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:00.188207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:00.188863 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:00.189142 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:00.189414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] PATH_INFO: `/v3/users/83f3e29a1f304013ac3b451b8fcd54fb` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:00.190612 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] RBAC: Authorizing `identity:delete_user(user_id=83f3e29a1f304013ac3b451b8fcd54fb)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:00.192531 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:00.434590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '83f3e29a1f304013ac3b451b8fcd54fb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:00.435270 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '83f3e29a1f304013ac3b451b8fcd54fb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:00.443653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '83f3e29a1f304013ac3b451b8fcd54fb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:00.444412 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '83f3e29a1f304013ac3b451b8fcd54fb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:00.446548 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:00.520773 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '83f3e29a1f304013ac3b451b8fcd54fb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:00.541835 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1374/2738] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:59:00 2022] DELETE /identity/v3/users/83f3e29a1f304013ac3b451b8fcd54fb => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:00.546375 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694db603-4f5d-40cf-99f4-ea50cfafea42 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:00.546660 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694db603-4f5d-40cf-99f4-ea50cfafea42 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:00.550195 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694db603-4f5d-40cf-99f4-ea50cfafea42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:00.566326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-694db603-4f5d-40cf-99f4-ea50cfafea42 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:00.567696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-694db603-4f5d-40cf-99f4-ea50cfafea42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:00.645983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1365/2739] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:00.659754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:00.661219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:00.740809 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5607db8-c942-44f0-b1c8-7a60d1341979 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:00.744555 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e413d4b9-168e-400f-8473-82fd6b520d2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:00.745325 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e413d4b9-168e-400f-8473-82fd6b520d2c service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:00.745611 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e413d4b9-168e-400f-8473-82fd6b520d2c service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:00.745887 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e413d4b9-168e-400f-8473-82fd6b520d2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:00.755803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e413d4b9-168e-400f-8473-82fd6b520d2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:00.759846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e413d4b9-168e-400f-8473-82fd6b520d2c service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:00.772879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1375/2740] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:59:01.354777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694db603-4f5d-40cf-99f4-ea50cfafea42 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:01.356863 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-694db603-4f5d-40cf-99f4-ea50cfafea42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:01.374111 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694db603-4f5d-40cf-99f4-ea50cfafea42 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:01.381176 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710b9702-4a38-4656-a507-98c6b6345318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:01.383210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710b9702-4a38-4656-a507-98c6b6345318 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:01.383787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710b9702-4a38-4656-a507-98c6b6345318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:01.384294 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710b9702-4a38-4656-a507-98c6b6345318 admin admin] PATH_INFO: `/v3/projects/2be393c7a74a45828c2086f843aa1486` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:01.393764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710b9702-4a38-4656-a507-98c6b6345318 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2be393c7a74a45828c2086f843aa1486)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:01.397574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710b9702-4a38-4656-a507-98c6b6345318 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:01.415088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:01.535870 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-710b9702-4a38-4656-a507-98c6b6345318 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2be393c7a74a45828c2086f843aa1486', 'request_id': 'req-710b9702-4a38-4656-a507-98c6b6345318'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:01.558758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-710b9702-4a38-4656-a507-98c6b6345318 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2be393c7a74a45828c2086f843aa1486', 'request_id': 'req-710b9702-4a38-4656-a507-98c6b6345318'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:01.576823 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-710b9702-4a38-4656-a507-98c6b6345318 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2be393c7a74a45828c2086f843aa1486', 'request_id': 'req-710b9702-4a38-4656-a507-98c6b6345318'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:01.652502 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-710b9702-4a38-4656-a507-98c6b6345318 admin admin] The token cache is being invalidate because project 2be393c7a74a45828c2086f843aa1486 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:59:01.653592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-710b9702-4a38-4656-a507-98c6b6345318 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:01.656594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-710b9702-4a38-4656-a507-98c6b6345318 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:01.658184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1366/2741] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:59:01 2022] DELETE /identity/v3/projects/2be393c7a74a45828c2086f843aa1486 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:01.663282 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeb765d-f105-4fcb-8eff-b1091bdbd81e None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:01.664463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeb765d-f105-4fcb-8eff-b1091bdbd81e None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:01.665943 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeb765d-f105-4fcb-8eff-b1091bdbd81e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:01.688411 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-9aeb765d-f105-4fcb-8eff-b1091bdbd81e None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:01.689573 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9aeb765d-f105-4fcb-8eff-b1091bdbd81e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:01.759171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1376/2742] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:01.767420 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27992aae-7961-4a5e-9d12-e96b81803f48 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:01.767801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27992aae-7961-4a5e-9d12-e96b81803f48 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:01.768079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27992aae-7961-4a5e-9d12-e96b81803f48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:01.788894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-27992aae-7961-4a5e-9d12-e96b81803f48 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:01.790658 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-27992aae-7961-4a5e-9d12-e96b81803f48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:01.803500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1367/2743] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:01.807944 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeb765d-f105-4fcb-8eff-b1091bdbd81e None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:01.818094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeb765d-f105-4fcb-8eff-b1091bdbd81e None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:01.824694 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a616b36c-ed34-461f-8a41-e557a71026dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:01.825331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a616b36c-ed34-461f-8a41-e557a71026dc admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:01.825610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a616b36c-ed34-461f-8a41-e557a71026dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:01.825876 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a616b36c-ed34-461f-8a41-e557a71026dc admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:01.826594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a616b36c-ed34-461f-8a41-e557a71026dc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:59:01.826979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a616b36c-ed34-461f-8a41-e557a71026dc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:01.828778 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a616b36c-ed34-461f-8a41-e557a71026dc admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:01.845710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1377/2744] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:59:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:59:01.850265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27992aae-7961-4a5e-9d12-e96b81803f48 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:01.861348 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27992aae-7961-4a5e-9d12-e96b81803f48 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:01.866429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18d90ac-5fae-422d-8cb9-98f5c20f612f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:01.868603 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18d90ac-5fae-422d-8cb9-98f5c20f612f admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:01.868896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18d90ac-5fae-422d-8cb9-98f5c20f612f admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:01.869170 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18d90ac-5fae-422d-8cb9-98f5c20f612f admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:01.869809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18d90ac-5fae-422d-8cb9-98f5c20f612f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:01.871759 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18d90ac-5fae-422d-8cb9-98f5c20f612f admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:01.981748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1368/2745] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:59:01 2022] POST /identity/v3/projects => generated 377 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:59:01.985879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a616b36c-ed34-461f-8a41-e557a71026dc admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:01.996904 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a616b36c-ed34-461f-8a41-e557a71026dc admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:02.058336 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294094d9-d0d3-45c6-99e7-21130e74db32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:02.058983 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294094d9-d0d3-45c6-99e7-21130e74db32 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:02.059302 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294094d9-d0d3-45c6-99e7-21130e74db32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:02.059582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294094d9-d0d3-45c6-99e7-21130e74db32 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:02.060231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294094d9-d0d3-45c6-99e7-21130e74db32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:02.062281 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294094d9-d0d3-45c6-99e7-21130e74db32 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:02.074162 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:INSERT=3,keystone:DELETE=3,keystone:SELECT=94 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:02.170085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1378/2746] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:59:01 2022] POST /identity/v3/users => generated 289 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:59:02.171506 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18d90ac-5fae-422d-8cb9-98f5c20f612f admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:02.183039 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18d90ac-5fae-422d-8cb9-98f5c20f612f admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:02.187819 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89660044-ae79-46f6-874a-273521d51eef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:02.188423 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89660044-ae79-46f6-874a-273521d51eef admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:02.188705 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89660044-ae79-46f6-874a-273521d51eef admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:02.188983 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89660044-ae79-46f6-874a-273521d51eef admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:02.189628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89660044-ae79-46f6-874a-273521d51eef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:02.191260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89660044-ae79-46f6-874a-273521d51eef admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:02.203234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1369/2747] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:02 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:59:02.208151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294094d9-d0d3-45c6-99e7-21130e74db32 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:02.218501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294094d9-d0d3-45c6-99e7-21130e74db32 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:02.223159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32a1856-efd6-4565-b1db-009c2fff0db2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:02.223752 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32a1856-efd6-4565-b1db-009c2fff0db2 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:02.224031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32a1856-efd6-4565-b1db-009c2fff0db2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:02.224300 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32a1856-efd6-4565-b1db-009c2fff0db2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:02.224941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32a1856-efd6-4565-b1db-009c2fff0db2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:02.227439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32a1856-efd6-4565-b1db-009c2fff0db2 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:02.237924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1379/2748] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:02 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:59:02.242913 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89660044-ae79-46f6-874a-273521d51eef admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:02.253234 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89660044-ae79-46f6-874a-273521d51eef admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:02.258781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379b51b6-27dd-4571-95d9-ca90f6bf0b07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:02.259444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379b51b6-27dd-4571-95d9-ca90f6bf0b07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:02.260496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379b51b6-27dd-4571-95d9-ca90f6bf0b07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:02.260496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379b51b6-27dd-4571-95d9-ca90f6bf0b07 admin admin] PATH_INFO: `/v3/projects/bd69fc15eb864544a327441cc196eddc/users/fb84a5e40d78497aa1b8658b6962e349/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:02.275158 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379b51b6-27dd-4571-95d9-ca90f6bf0b07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd69fc15eb864544a327441cc196eddc, user_id=fb84a5e40d78497aa1b8658b6962e349, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:02.277808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379b51b6-27dd-4571-95d9-ca90f6bf0b07 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:02.343668 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1370/2749] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:59:02 2022] PUT /identity/v3/projects/bd69fc15eb864544a327441cc196eddc/users/fb84a5e40d78497aa1b8658b6962e349/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:02.355962 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdc0ab7-45f4-45c1-a628-e98961d3229a None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:02.356249 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdc0ab7-45f4-45c1-a628-e98961d3229a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:02.356508 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdc0ab7-45f4-45c1-a628-e98961d3229a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:02.385480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-1fdc0ab7-45f4-45c1-a628-e98961d3229a None None] MFA Rules not processed for user `fb84a5e40d78497aa1b8658b6962e349`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:02.387039 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1fdc0ab7-45f4-45c1-a628-e98961d3229a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:02.458142 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1380/2750] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:02 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:59:02.469836 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379b51b6-27dd-4571-95d9-ca90f6bf0b07 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:02.471574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-379b51b6-27dd-4571-95d9-ca90f6bf0b07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:02.531179 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379b51b6-27dd-4571-95d9-ca90f6bf0b07 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:02.534926 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd4c35b-aeec-4df7-aea8-6a09f0f4f84f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:02.535528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd4c35b-aeec-4df7-aea8-6a09f0f4f84f service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:02.535808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd4c35b-aeec-4df7-aea8-6a09f0f4f84f service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:02.536148 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd4c35b-aeec-4df7-aea8-6a09f0f4f84f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:02.548369 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd4c35b-aeec-4df7-aea8-6a09f0f4f84f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:02.550012 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd4c35b-aeec-4df7-aea8-6a09f0f4f84f service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:02.560095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1371/2751] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:02 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:12.455155 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:INSERT=1,keystone:SELECT=39 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:12.555925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=72,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:13.385658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdc0ab7-45f4-45c1-a628-e98961d3229a None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:13.387909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1fdc0ab7-45f4-45c1-a628-e98961d3229a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:13.452293 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdc0ab7-45f4-45c1-a628-e98961d3229a None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:13.456989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:13.457623 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:13.457904 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:13.458208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] PATH_INFO: `/v3/users/d769540a992b4eacb8e6a4aff286581c` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:13.459396 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=d769540a992b4eacb8e6a4aff286581c)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:13.461303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:13.671151 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd769540a992b4eacb8e6a4aff286581c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:13.671839 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd769540a992b4eacb8e6a4aff286581c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:13.678936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd769540a992b4eacb8e6a4aff286581c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:13.679579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd769540a992b4eacb8e6a4aff286581c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:13.828852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd769540a992b4eacb8e6a4aff286581c'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:13.847524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1381/2752] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:59:13 2022] DELETE /identity/v3/users/d769540a992b4eacb8e6a4aff286581c => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:13.852094 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47c1d69-0b3a-4f06-8189-db7944ddc1f2 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:13.852586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47c1d69-0b3a-4f06-8189-db7944ddc1f2 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:13.852694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47c1d69-0b3a-4f06-8189-db7944ddc1f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:13.872600 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d47c1d69-0b3a-4f06-8189-db7944ddc1f2 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:13.873732 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d47c1d69-0b3a-4f06-8189-db7944ddc1f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:13.932445 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1372/2753] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:13.941935 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:13.943395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:13.999814 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d304c7a1-0a6e-4e50-93b0-f915a92be2cf admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:14.003484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943bfebe-989c-4ea0-b67b-854ed95ab819 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:14.004083 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943bfebe-989c-4ea0-b67b-854ed95ab819 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:14.004361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943bfebe-989c-4ea0-b67b-854ed95ab819 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:14.004636 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943bfebe-989c-4ea0-b67b-854ed95ab819 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:14.014263 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943bfebe-989c-4ea0-b67b-854ed95ab819 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:14.015724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943bfebe-989c-4ea0-b67b-854ed95ab819 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:14.025070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1382/2754] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:14.487706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47c1d69-0b3a-4f06-8189-db7944ddc1f2 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:14.489128 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d47c1d69-0b3a-4f06-8189-db7944ddc1f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:14.502976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47c1d69-0b3a-4f06-8189-db7944ddc1f2 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:14.507877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:14.508586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:14.508860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:14.509249 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] PATH_INFO: `/v3/projects/adfea6e2a27e49559d531c8e0f131d2a` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:14.510724 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] RBAC: Authorizing `identity:delete_project(project_id=adfea6e2a27e49559d531c8e0f131d2a)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:14.512714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:14.681775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'adfea6e2a27e49559d531c8e0f131d2a', 'request_id': 'req-34db5fcd-7a73-484c-9a3e-9ca947659f0c'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:14.687153 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'adfea6e2a27e49559d531c8e0f131d2a', 'request_id': 'req-34db5fcd-7a73-484c-9a3e-9ca947659f0c'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:14.693352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'adfea6e2a27e49559d531c8e0f131d2a', 'request_id': 'req-34db5fcd-7a73-484c-9a3e-9ca947659f0c'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:14.765361 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] The token cache is being invalidate because project adfea6e2a27e49559d531c8e0f131d2a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:59:14.765573 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:14.766318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:14.767981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1373/2755] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:59:14 2022] DELETE /identity/v3/projects/adfea6e2a27e49559d531c8e0f131d2a => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:14.772212 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943bfebe-989c-4ea0-b67b-854ed95ab819 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:14.782950 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943bfebe-989c-4ea0-b67b-854ed95ab819 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:14.801677 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:14.840372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b154bf-bf63-4eb3-bcdb-f7bde099cd70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:14.840986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b154bf-bf63-4eb3-bcdb-f7bde099cd70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:14.841260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b154bf-bf63-4eb3-bcdb-f7bde099cd70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:14.841971 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b154bf-bf63-4eb3-bcdb-f7bde099cd70 admin admin] PATH_INFO: `/v3/users/5114f45f4ae14962a38d5bfe85e5dde7` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:14.842747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b154bf-bf63-4eb3-bcdb-f7bde099cd70 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5114f45f4ae14962a38d5bfe85e5dde7)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:14.844724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b154bf-bf63-4eb3-bcdb-f7bde099cd70 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:15.001450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-13b154bf-bf63-4eb3-bcdb-f7bde099cd70 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5114f45f4ae14962a38d5bfe85e5dde7'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:15.002107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-13b154bf-bf63-4eb3-bcdb-f7bde099cd70 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5114f45f4ae14962a38d5bfe85e5dde7'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:15.008948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-13b154bf-bf63-4eb3-bcdb-f7bde099cd70 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5114f45f4ae14962a38d5bfe85e5dde7'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:15.009604 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-13b154bf-bf63-4eb3-bcdb-f7bde099cd70 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5114f45f4ae14962a38d5bfe85e5dde7'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:15.079599 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-13b154bf-bf63-4eb3-bcdb-f7bde099cd70 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5114f45f4ae14962a38d5bfe85e5dde7'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:15.097491 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1383/2756] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:59:14 2022] DELETE /identity/v3/users/5114f45f4ae14962a38d5bfe85e5dde7 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:15.527592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:15.529117 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:15.544343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34db5fcd-7a73-484c-9a3e-9ca947659f0c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:15.609503 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8bb648-cac1-424b-94d5-bce95e9037e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:15.610224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8bb648-cac1-424b-94d5-bce95e9037e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:15.610504 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8bb648-cac1-424b-94d5-bce95e9037e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:15.610769 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8bb648-cac1-424b-94d5-bce95e9037e1 admin admin] PATH_INFO: `/v3/projects/7b363b6ea1ae4859b3d682deca8bbd9c` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:15.611907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8bb648-cac1-424b-94d5-bce95e9037e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b363b6ea1ae4859b3d682deca8bbd9c)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:15.613854 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8bb648-cac1-424b-94d5-bce95e9037e1 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:15.721954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:15.739496 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9d8bb648-cac1-424b-94d5-bce95e9037e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b363b6ea1ae4859b3d682deca8bbd9c', 'request_id': 'req-9d8bb648-cac1-424b-94d5-bce95e9037e1'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:15.746379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9d8bb648-cac1-424b-94d5-bce95e9037e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b363b6ea1ae4859b3d682deca8bbd9c', 'request_id': 'req-9d8bb648-cac1-424b-94d5-bce95e9037e1'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:15.752586 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9d8bb648-cac1-424b-94d5-bce95e9037e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b363b6ea1ae4859b3d682deca8bbd9c', 'request_id': 'req-9d8bb648-cac1-424b-94d5-bce95e9037e1'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:15.805447 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9d8bb648-cac1-424b-94d5-bce95e9037e1 admin admin] The token cache is being invalidate because project 7b363b6ea1ae4859b3d682deca8bbd9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:59:15.805736 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9d8bb648-cac1-424b-94d5-bce95e9037e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:15.806456 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9d8bb648-cac1-424b-94d5-bce95e9037e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:15.808699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1374/2757] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:59:15 2022] DELETE /identity/v3/projects/7b363b6ea1ae4859b3d682deca8bbd9c => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:15.813455 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d07c5c2-1b1b-48d3-824f-8494130c435d None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:15.813993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d07c5c2-1b1b-48d3-824f-8494130c435d None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:15.813993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d07c5c2-1b1b-48d3-824f-8494130c435d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:15.833769 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-4d07c5c2-1b1b-48d3-824f-8494130c435d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:15.835029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-4d07c5c2-1b1b-48d3-824f-8494130c435d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:15.897610 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1384/2758] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:15.905734 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0c0af6-a28c-4c63-8769-c16bb77ba50b None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:15.906036 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0c0af6-a28c-4c63-8769-c16bb77ba50b None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:15.906326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0c0af6-a28c-4c63-8769-c16bb77ba50b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:15.927434 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-6f0c0af6-a28c-4c63-8769-c16bb77ba50b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:15.928742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-6f0c0af6-a28c-4c63-8769-c16bb77ba50b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:15.938281 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1375/2759] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:15.942856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d07c5c2-1b1b-48d3-824f-8494130c435d None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:15.953076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d07c5c2-1b1b-48d3-824f-8494130c435d None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:15.957955 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f18fe75-1e1b-4f73-8081-cec197884066 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:15.958591 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f18fe75-1e1b-4f73-8081-cec197884066 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:15.958873 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f18fe75-1e1b-4f73-8081-cec197884066 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:15.959148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f18fe75-1e1b-4f73-8081-cec197884066 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:15.959796 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f18fe75-1e1b-4f73-8081-cec197884066 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:59:15.960100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f18fe75-1e1b-4f73-8081-cec197884066 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:15.961735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f18fe75-1e1b-4f73-8081-cec197884066 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:15.973587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1385/2760] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:59:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:59:15.978185 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0c0af6-a28c-4c63-8769-c16bb77ba50b None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:15.988134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0c0af6-a28c-4c63-8769-c16bb77ba50b None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:15.992827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f48312a-cb71-4aa8-b16f-3d4c22a1b20a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:15.993418 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f48312a-cb71-4aa8-b16f-3d4c22a1b20a admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:15.993689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f48312a-cb71-4aa8-b16f-3d4c22a1b20a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:15.993955 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f48312a-cb71-4aa8-b16f-3d4c22a1b20a admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:15.995381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f48312a-cb71-4aa8-b16f-3d4c22a1b20a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:15.997343 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f48312a-cb71-4aa8-b16f-3d4c22a1b20a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:16.081597 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1376/2761] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:59:15 2022] POST /identity/v3/projects => generated 363 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:59:16.086079 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f18fe75-1e1b-4f73-8081-cec197884066 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:16.096768 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f18fe75-1e1b-4f73-8081-cec197884066 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:16.124364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:16.154260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6136ef-4740-420e-b50c-a939fa3e258e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:16.154793 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6136ef-4740-420e-b50c-a939fa3e258e admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:16.155072 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6136ef-4740-420e-b50c-a939fa3e258e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:16.155344 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6136ef-4740-420e-b50c-a939fa3e258e admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:16.155972 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6136ef-4740-420e-b50c-a939fa3e258e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:16.157888 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6136ef-4740-420e-b50c-a939fa3e258e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:16.268316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1386/2762] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:59:16 2022] POST /identity/v3/users => generated 282 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:59:16.272633 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f48312a-cb71-4aa8-b16f-3d4c22a1b20a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:16.283673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f48312a-cb71-4aa8-b16f-3d4c22a1b20a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:16.288350 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c63ebb-bdb4-4386-a616-f91cd64eef8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:16.288939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c63ebb-bdb4-4386-a616-f91cd64eef8c admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:16.289216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c63ebb-bdb4-4386-a616-f91cd64eef8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:16.289490 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c63ebb-bdb4-4386-a616-f91cd64eef8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:16.290175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c63ebb-bdb4-4386-a616-f91cd64eef8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:16.292392 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c63ebb-bdb4-4386-a616-f91cd64eef8c admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:16.302378 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1377/2763] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:59:16.306353 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6136ef-4740-420e-b50c-a939fa3e258e admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:16.316469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6136ef-4740-420e-b50c-a939fa3e258e admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:16.321148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ff0833-cbeb-4641-9db9-b1a5a1f10cfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:16.321741 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ff0833-cbeb-4641-9db9-b1a5a1f10cfb admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:16.322040 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ff0833-cbeb-4641-9db9-b1a5a1f10cfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:16.322339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ff0833-cbeb-4641-9db9-b1a5a1f10cfb admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:16.322982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ff0833-cbeb-4641-9db9-b1a5a1f10cfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:16.324589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ff0833-cbeb-4641-9db9-b1a5a1f10cfb admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:16.334958 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1387/2764] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:16 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:59:16.339778 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c63ebb-bdb4-4386-a616-f91cd64eef8c admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:16.349795 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c63ebb-bdb4-4386-a616-f91cd64eef8c admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:16.355498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13da6781-6e9e-4b1c-96bb-6dce45b9240f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:16.356133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13da6781-6e9e-4b1c-96bb-6dce45b9240f admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:16.356408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13da6781-6e9e-4b1c-96bb-6dce45b9240f admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:16.356673 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13da6781-6e9e-4b1c-96bb-6dce45b9240f admin admin] PATH_INFO: `/v3/projects/de3d71f3f5de4255a8d76a11038b0a68/users/023b76bda7fd4743913fecc123d77e02/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:16.371934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13da6781-6e9e-4b1c-96bb-6dce45b9240f admin admin] RBAC: Authorizing `identity:create_grant(project_id=de3d71f3f5de4255a8d76a11038b0a68, user_id=023b76bda7fd4743913fecc123d77e02, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:16.381409 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13da6781-6e9e-4b1c-96bb-6dce45b9240f admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:16.474407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1378/2765] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:59:16 2022] PUT /identity/v3/projects/de3d71f3f5de4255a8d76a11038b0a68/users/023b76bda7fd4743913fecc123d77e02/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:16.485118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f29ac2-8599-45bc-a08c-5aaf0ce861ff None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:16.485406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f29ac2-8599-45bc-a08c-5aaf0ce861ff None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:16.485683 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f29ac2-8599-45bc-a08c-5aaf0ce861ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:16.513513 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-55f29ac2-8599-45bc-a08c-5aaf0ce861ff None None] MFA Rules not processed for user `023b76bda7fd4743913fecc123d77e02`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:16.514796 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-55f29ac2-8599-45bc-a08c-5aaf0ce861ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:16.579202 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1388/2766] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:16 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:16.583879 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13da6781-6e9e-4b1c-96bb-6dce45b9240f admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:16.646588 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13da6781-6e9e-4b1c-96bb-6dce45b9240f admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:16.652022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c647dd-774e-448f-8405-3297f62b034e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:16.652619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c647dd-774e-448f-8405-3297f62b034e admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:16.652900 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c647dd-774e-448f-8405-3297f62b034e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:16.653175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c647dd-774e-448f-8405-3297f62b034e admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:16.653806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c647dd-774e-448f-8405-3297f62b034e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:16.655749 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c647dd-774e-448f-8405-3297f62b034e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:16.751321 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1379/2767] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:59:16 2022] POST /identity/v3/projects => generated 363 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:59:16.755634 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f29ac2-8599-45bc-a08c-5aaf0ce861ff None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:16.767453 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f29ac2-8599-45bc-a08c-5aaf0ce861ff None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:16.825363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d03f8e3-35b0-4082-95ca-d7969fcacacb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:16.826213 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d03f8e3-35b0-4082-95ca-d7969fcacacb admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:16.826366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d03f8e3-35b0-4082-95ca-d7969fcacacb admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:16.826579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d03f8e3-35b0-4082-95ca-d7969fcacacb admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:16.827203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d03f8e3-35b0-4082-95ca-d7969fcacacb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:16.829233 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d03f8e3-35b0-4082-95ca-d7969fcacacb admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:16.938845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1389/2768] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:59:16 2022] POST /identity/v3/users => generated 288 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:59:16.942401 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c647dd-774e-448f-8405-3297f62b034e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:16.953581 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c647dd-774e-448f-8405-3297f62b034e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:16.958143 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7445b9b0-e440-4e69-85f4-2ac55ac8c4f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:16.958766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7445b9b0-e440-4e69-85f4-2ac55ac8c4f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:16.959056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7445b9b0-e440-4e69-85f4-2ac55ac8c4f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:16.959326 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7445b9b0-e440-4e69-85f4-2ac55ac8c4f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:16.959965 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7445b9b0-e440-4e69-85f4-2ac55ac8c4f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:16.961551 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7445b9b0-e440-4e69-85f4-2ac55ac8c4f2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:16.972420 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1380/2769] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:59:16.977262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d03f8e3-35b0-4082-95ca-d7969fcacacb admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:16.989617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d03f8e3-35b0-4082-95ca-d7969fcacacb admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:16.995067 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e360a8b-989d-4e90-a4eb-6e0aa26aee38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:16.995908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e360a8b-989d-4e90-a4eb-6e0aa26aee38 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:16.996188 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e360a8b-989d-4e90-a4eb-6e0aa26aee38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:16.996464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e360a8b-989d-4e90-a4eb-6e0aa26aee38 admin admin] PATH_INFO: `/v3/projects/77c171a67f1c4cc782bc42bdb8d3b5c6/users/cc68bc97c8d449dcb878ee38d61a56b3/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:17.012642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e360a8b-989d-4e90-a4eb-6e0aa26aee38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77c171a67f1c4cc782bc42bdb8d3b5c6, user_id=cc68bc97c8d449dcb878ee38d61a56b3, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:17.015339 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e360a8b-989d-4e90-a4eb-6e0aa26aee38 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:17.087003 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1390/2770] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:59:16 2022] PUT /identity/v3/projects/77c171a67f1c4cc782bc42bdb8d3b5c6/users/cc68bc97c8d449dcb878ee38d61a56b3/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:17.091062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7445b9b0-e440-4e69-85f4-2ac55ac8c4f2 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:17.103837 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7445b9b0-e440-4e69-85f4-2ac55ac8c4f2 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:17.127272 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:17.167575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d7d53b-2dfa-471d-b889-729a950bafeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:17.168183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d7d53b-2dfa-471d-b889-729a950bafeb admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:17.168467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d7d53b-2dfa-471d-b889-729a950bafeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:17.168738 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d7d53b-2dfa-471d-b889-729a950bafeb admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:17.169408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d7d53b-2dfa-471d-b889-729a950bafeb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:17.171065 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d7d53b-2dfa-471d-b889-729a950bafeb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:17.181260 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1381/2771] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:17 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:59:17.186280 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e360a8b-989d-4e90-a4eb-6e0aa26aee38 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:17.192590 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:17.197544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e360a8b-989d-4e90-a4eb-6e0aa26aee38 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:17.202465 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01db275e-838c-45cc-a293-aa33a0b975a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:17.203129 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01db275e-838c-45cc-a293-aa33a0b975a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:17.203406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01db275e-838c-45cc-a293-aa33a0b975a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:17.203676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01db275e-838c-45cc-a293-aa33a0b975a8 admin admin] PATH_INFO: `/v3/projects/77c171a67f1c4cc782bc42bdb8d3b5c6/users/cc68bc97c8d449dcb878ee38d61a56b3/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:17.205724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01db275e-838c-45cc-a293-aa33a0b975a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77c171a67f1c4cc782bc42bdb8d3b5c6, user_id=cc68bc97c8d449dcb878ee38d61a56b3, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:17.208385 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01db275e-838c-45cc-a293-aa33a0b975a8 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:17.219386 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1391/2772] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:59:17 2022] PUT /identity/v3/projects/77c171a67f1c4cc782bc42bdb8d3b5c6/users/cc68bc97c8d449dcb878ee38d61a56b3/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:59:17.231625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd6136a-942e-44ed-9562-71670fa7df66 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:17.231907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd6136a-942e-44ed-9562-71670fa7df66 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:17.232186 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd6136a-942e-44ed-9562-71670fa7df66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:17.262441 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-0dd6136a-942e-44ed-9562-71670fa7df66 None None] MFA Rules not processed for user `cc68bc97c8d449dcb878ee38d61a56b3`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:17.263697 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0dd6136a-942e-44ed-9562-71670fa7df66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:17.336337 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1382/2773] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:17 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:17.347577 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01db275e-838c-45cc-a293-aa33a0b975a8 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:17.349312 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-01db275e-838c-45cc-a293-aa33a0b975a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:17.361067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd6136a-942e-44ed-9562-71670fa7df66 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:17.362542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0dd6136a-942e-44ed-9562-71670fa7df66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:17.409829 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01db275e-838c-45cc-a293-aa33a0b975a8 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:17.413609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5455241c-3b83-40df-806a-b98bbd2018c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:17.414255 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5455241c-3b83-40df-806a-b98bbd2018c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:17.414534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5455241c-3b83-40df-806a-b98bbd2018c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:17.414800 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5455241c-3b83-40df-806a-b98bbd2018c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:17.425206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5455241c-3b83-40df-806a-b98bbd2018c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:17.426584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd6136a-942e-44ed-9562-71670fa7df66 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:17.426756 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5455241c-3b83-40df-806a-b98bbd2018c3 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:17.430450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b027a5-50d6-424b-9e6b-7f7f485e641e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:17.431214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b027a5-50d6-424b-9e6b-7f7f485e641e service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:17.434512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b027a5-50d6-424b-9e6b-7f7f485e641e service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:17.434512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b027a5-50d6-424b-9e6b-7f7f485e641e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:17.434909 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-e9b027a5-50d6-424b-9e6b-7f7f485e641e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:17.436934 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1392/2774] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:17 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:17.445062 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b027a5-50d6-424b-9e6b-7f7f485e641e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:17.446693 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b027a5-50d6-424b-9e6b-7f7f485e641e service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:17.455291 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1383/2775] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:59:17.649776 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5455241c-3b83-40df-806a-b98bbd2018c3 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:17.657739 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5455241c-3b83-40df-806a-b98bbd2018c3 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:17.661516 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d399b6be-9ef3-4abc-ac87-0ff7e79d6979 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:17.662112 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d399b6be-9ef3-4abc-ac87-0ff7e79d6979 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:17.662483 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d399b6be-9ef3-4abc-ac87-0ff7e79d6979 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:17.662762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d399b6be-9ef3-4abc-ac87-0ff7e79d6979 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:17.723710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d399b6be-9ef3-4abc-ac87-0ff7e79d6979 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:17.725171 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d399b6be-9ef3-4abc-ac87-0ff7e79d6979 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:17.735542 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1393/2776] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:17 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:18.210463 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b027a5-50d6-424b-9e6b-7f7f485e641e service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:18.212002 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-e9b027a5-50d6-424b-9e6b-7f7f485e641e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:18.267848 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:18.288834 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b027a5-50d6-424b-9e6b-7f7f485e641e service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:18.293892 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d452fd-6724-4d97-bbe3-0509bc77bccd service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:18.294556 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d452fd-6724-4d97-bbe3-0509bc77bccd service placement] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:18.294831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d452fd-6724-4d97-bbe3-0509bc77bccd service placement] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:18.295099 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d452fd-6724-4d97-bbe3-0509bc77bccd service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:18.296801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a2d452fd-6724-4d97-bbe3-0509bc77bccd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:18.308818 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d452fd-6724-4d97-bbe3-0509bc77bccd service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:18.310364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d452fd-6724-4d97-bbe3-0509bc77bccd service placement] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:18.320584 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1384/2777] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:21.306299 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d399b6be-9ef3-4abc-ac87-0ff7e79d6979 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:21.387628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d399b6be-9ef3-4abc-ac87-0ff7e79d6979 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:21.392954 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:21.393641 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:21.393859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:21.394130 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] PATH_INFO: `/v3/users/023b76bda7fd4743913fecc123d77e02` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:21.395449 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] RBAC: Authorizing `identity:delete_user(user_id=023b76bda7fd4743913fecc123d77e02)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:21.397544 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:21.574285 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:21.598065 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '023b76bda7fd4743913fecc123d77e02'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:21.599081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '023b76bda7fd4743913fecc123d77e02'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:21.606906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '023b76bda7fd4743913fecc123d77e02'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:21.607669 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '023b76bda7fd4743913fecc123d77e02'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:21.674825 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '023b76bda7fd4743913fecc123d77e02'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:21.696261 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1394/2778] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:59:21 2022] DELETE /identity/v3/users/023b76bda7fd4743913fecc123d77e02 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:21.701385 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5f27a7-ba09-4f52-9468-a4d0c9a14326 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:21.701922 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5f27a7-ba09-4f52-9468-a4d0c9a14326 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:21.702042 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5f27a7-ba09-4f52-9468-a4d0c9a14326 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:21.722109 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-5e5f27a7-ba09-4f52-9468-a4d0c9a14326 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:21.723383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5e5f27a7-ba09-4f52-9468-a4d0c9a14326 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:21.786127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1385/2779] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:59:21.798330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:21.799777 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:21.862698 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09f9099-5f8e-46e3-9b59-882be1d21525 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:21.866875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1125a4e7-8fee-466d-b61f-c6a9f9e7b205 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:21.867515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1125a4e7-8fee-466d-b61f-c6a9f9e7b205 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:21.867760 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1125a4e7-8fee-466d-b61f-c6a9f9e7b205 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:21.868030 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1125a4e7-8fee-466d-b61f-c6a9f9e7b205 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:21.878064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1125a4e7-8fee-466d-b61f-c6a9f9e7b205 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:21.879532 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1125a4e7-8fee-466d-b61f-c6a9f9e7b205 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:21.889225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1395/2780] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:22.441387 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5f27a7-ba09-4f52-9468-a4d0c9a14326 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:22.443053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5e5f27a7-ba09-4f52-9468-a4d0c9a14326 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:22.457945 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5f27a7-ba09-4f52-9468-a4d0c9a14326 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:22.462025 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:22.462695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:22.462975 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:22.463407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] PATH_INFO: `/v3/projects/de3d71f3f5de4255a8d76a11038b0a68` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:22.464895 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=de3d71f3f5de4255a8d76a11038b0a68)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:22.466851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:22.600587 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de3d71f3f5de4255a8d76a11038b0a68', 'request_id': 'req-4839c871-297c-4750-a9f8-fa3ba11550f6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:22.609634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de3d71f3f5de4255a8d76a11038b0a68', 'request_id': 'req-4839c871-297c-4750-a9f8-fa3ba11550f6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:22.615934 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de3d71f3f5de4255a8d76a11038b0a68', 'request_id': 'req-4839c871-297c-4750-a9f8-fa3ba11550f6'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:22.691411 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] The token cache is being invalidate because project de3d71f3f5de4255a8d76a11038b0a68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:59:22.691706 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:22.692338 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:22.693815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1386/2781] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:59:22 2022] DELETE /identity/v3/projects/de3d71f3f5de4255a8d76a11038b0a68 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:22.698475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1125a4e7-8fee-466d-b61f-c6a9f9e7b205 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:22.708813 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1125a4e7-8fee-466d-b61f-c6a9f9e7b205 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:22.769286 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab354ade-8c68-4156-ae5e-f42a61e35dfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:22.769912 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab354ade-8c68-4156-ae5e-f42a61e35dfe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:22.770206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab354ade-8c68-4156-ae5e-f42a61e35dfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:22.770477 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab354ade-8c68-4156-ae5e-f42a61e35dfe admin admin] PATH_INFO: `/v3/users/cc68bc97c8d449dcb878ee38d61a56b3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:22.771655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab354ade-8c68-4156-ae5e-f42a61e35dfe admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc68bc97c8d449dcb878ee38d61a56b3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:22.773556 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab354ade-8c68-4156-ae5e-f42a61e35dfe admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:22.898875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ab354ade-8c68-4156-ae5e-f42a61e35dfe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc68bc97c8d449dcb878ee38d61a56b3'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:22.899524 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ab354ade-8c68-4156-ae5e-f42a61e35dfe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc68bc97c8d449dcb878ee38d61a56b3'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:22.905735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:22.908851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ab354ade-8c68-4156-ae5e-f42a61e35dfe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc68bc97c8d449dcb878ee38d61a56b3'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:22.908851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ab354ade-8c68-4156-ae5e-f42a61e35dfe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc68bc97c8d449dcb878ee38d61a56b3'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:22.977102 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ab354ade-8c68-4156-ae5e-f42a61e35dfe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc68bc97c8d449dcb878ee38d61a56b3'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:22.994994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1396/2782] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:59:22 2022] DELETE /identity/v3/users/cc68bc97c8d449dcb878ee38d61a56b3 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:23.424199 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:23.425667 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:23.439852 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4839c871-297c-4750-a9f8-fa3ba11550f6 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:23.503585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:23.504156 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:23.504432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:23.504951 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2 admin admin] PATH_INFO: `/v3/projects/77c171a67f1c4cc782bc42bdb8d3b5c6` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:23.506038 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=77c171a67f1c4cc782bc42bdb8d3b5c6)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:23.507969 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:23.514994 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:23.617509 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77c171a67f1c4cc782bc42bdb8d3b5c6', 'request_id': 'req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:23.623699 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77c171a67f1c4cc782bc42bdb8d3b5c6', 'request_id': 'req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:23.630000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77c171a67f1c4cc782bc42bdb8d3b5c6', 'request_id': 'req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:23.707146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2 admin admin] The token cache is being invalidate because project 77c171a67f1c4cc782bc42bdb8d3b5c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:59:23.707429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:23.708081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-b4bcffd9-96b7-4abf-ab55-05f17e9110b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:23.709702 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1387/2783] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:59:23 2022] DELETE /identity/v3/projects/77c171a67f1c4cc782bc42bdb8d3b5c6 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:23.715604 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e2bbb-88f7-4d7a-94be-b0dfa9b92c5b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:23.716097 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e2bbb-88f7-4d7a-94be-b0dfa9b92c5b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:23.716195 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e2bbb-88f7-4d7a-94be-b0dfa9b92c5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:23.737786 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-7a4e2bbb-88f7-4d7a-94be-b0dfa9b92c5b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:23.739246 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7a4e2bbb-88f7-4d7a-94be-b0dfa9b92c5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:23.802123 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1397/2784] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:23.810043 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0023e53-92fb-4343-b7ce-8e18e4fd6d82 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:23.810445 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0023e53-92fb-4343-b7ce-8e18e4fd6d82 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:23.810794 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0023e53-92fb-4343-b7ce-8e18e4fd6d82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:23.831271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-f0023e53-92fb-4343-b7ce-8e18e4fd6d82 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:23.832674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-f0023e53-92fb-4343-b7ce-8e18e4fd6d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:23.841663 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1388/2785] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:23.846068 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e2bbb-88f7-4d7a-94be-b0dfa9b92c5b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:23.857029 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e2bbb-88f7-4d7a-94be-b0dfa9b92c5b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:23.861811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eacdc5d-1c4c-4cbe-98f0-ae9038368f9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:23.862446 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eacdc5d-1c4c-4cbe-98f0-ae9038368f9f admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:23.862753 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eacdc5d-1c4c-4cbe-98f0-ae9038368f9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:23.863046 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eacdc5d-1c4c-4cbe-98f0-ae9038368f9f admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:23.863690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eacdc5d-1c4c-4cbe-98f0-ae9038368f9f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:59:23.863991 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eacdc5d-1c4c-4cbe-98f0-ae9038368f9f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:23.865587 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eacdc5d-1c4c-4cbe-98f0-ae9038368f9f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:23.877416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1398/2786] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:59:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:59:23.881777 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0023e53-92fb-4343-b7ce-8e18e4fd6d82 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:23.891887 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0023e53-92fb-4343-b7ce-8e18e4fd6d82 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:23.896631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bc71d7-40b9-4c5b-834d-94901dadeadd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:23.897232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bc71d7-40b9-4c5b-834d-94901dadeadd admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:23.897512 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bc71d7-40b9-4c5b-834d-94901dadeadd admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:23.897831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bc71d7-40b9-4c5b-834d-94901dadeadd admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:23.898439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9bc71d7-40b9-4c5b-834d-94901dadeadd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:23.900341 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9bc71d7-40b9-4c5b-834d-94901dadeadd admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:23.996066 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1389/2787] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:59:23 2022] POST /identity/v3/projects => generated 369 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:59:24.000423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eacdc5d-1c4c-4cbe-98f0-ae9038368f9f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:24.010989 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eacdc5d-1c4c-4cbe-98f0-ae9038368f9f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:24.068231 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e991acfc-b56f-432f-ac74-84e7bc424be7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:24.068754 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e991acfc-b56f-432f-ac74-84e7bc424be7 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:24.069037 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e991acfc-b56f-432f-ac74-84e7bc424be7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:24.069311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e991acfc-b56f-432f-ac74-84e7bc424be7 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:24.069932 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e991acfc-b56f-432f-ac74-84e7bc424be7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:24.071884 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e991acfc-b56f-432f-ac74-84e7bc424be7 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:24.171476 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1399/2788] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:59:23 2022] POST /identity/v3/users => generated 285 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:59:24.175615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bc71d7-40b9-4c5b-834d-94901dadeadd admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:24.185938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bc71d7-40b9-4c5b-834d-94901dadeadd admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:24.190636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85969bd7-a101-4da2-a694-b15b12964dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:24.191220 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85969bd7-a101-4da2-a694-b15b12964dd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:24.191524 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85969bd7-a101-4da2-a694-b15b12964dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:24.191799 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85969bd7-a101-4da2-a694-b15b12964dd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:24.192467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85969bd7-a101-4da2-a694-b15b12964dd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:24.194064 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85969bd7-a101-4da2-a694-b15b12964dd6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:24.204417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1390/2789] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:59:24.208316 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e991acfc-b56f-432f-ac74-84e7bc424be7 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:24.218994 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e991acfc-b56f-432f-ac74-84e7bc424be7 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:24.223744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666041a6-d717-49ef-b873-c086e3bb0ed3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:24.224331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666041a6-d717-49ef-b873-c086e3bb0ed3 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:24.224611 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666041a6-d717-49ef-b873-c086e3bb0ed3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:24.224884 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666041a6-d717-49ef-b873-c086e3bb0ed3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:24.225523 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666041a6-d717-49ef-b873-c086e3bb0ed3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:24.227447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666041a6-d717-49ef-b873-c086e3bb0ed3 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:24.238022 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1400/2790] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:59:24.242232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85969bd7-a101-4da2-a694-b15b12964dd6 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:24.252546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85969bd7-a101-4da2-a694-b15b12964dd6 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:24.257113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b3a39c-a6a2-48da-8ccb-738dd164c431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:24.257744 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b3a39c-a6a2-48da-8ccb-738dd164c431 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:24.258014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b3a39c-a6a2-48da-8ccb-738dd164c431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:24.258322 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b3a39c-a6a2-48da-8ccb-738dd164c431 admin admin] PATH_INFO: `/v3/projects/4dcd0d1ec0d3435e9b11b25df4200fc3/users/b90bad4106dc4de99194a0a966932e58/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:24.273539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b3a39c-a6a2-48da-8ccb-738dd164c431 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4dcd0d1ec0d3435e9b11b25df4200fc3, user_id=b90bad4106dc4de99194a0a966932e58, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:24.277127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b3a39c-a6a2-48da-8ccb-738dd164c431 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:24.331451 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1391/2791] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:59:24 2022] PUT /identity/v3/projects/4dcd0d1ec0d3435e9b11b25df4200fc3/users/b90bad4106dc4de99194a0a966932e58/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:59:24.342864 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9915e8cd-ab0c-4c27-b794-11a0be242577 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:24.343148 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9915e8cd-ab0c-4c27-b794-11a0be242577 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:24.343419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9915e8cd-ab0c-4c27-b794-11a0be242577 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:24.371623 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-9915e8cd-ab0c-4c27-b794-11a0be242577 None None] MFA Rules not processed for user `b90bad4106dc4de99194a0a966932e58`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:24.372832 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9915e8cd-ab0c-4c27-b794-11a0be242577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:24.398860 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:24.437646 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1401/2792] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:24 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:24.442134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b3a39c-a6a2-48da-8ccb-738dd164c431 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:24.507022 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b3a39c-a6a2-48da-8ccb-738dd164c431 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:24.512486 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74a6020-ab5c-45ac-9aa4-0678aa58e5d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:24.513088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74a6020-ab5c-45ac-9aa4-0678aa58e5d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:24.513364 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74a6020-ab5c-45ac-9aa4-0678aa58e5d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:24.513635 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74a6020-ab5c-45ac-9aa4-0678aa58e5d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:24.514306 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74a6020-ab5c-45ac-9aa4-0678aa58e5d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:24.516184 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74a6020-ab5c-45ac-9aa4-0678aa58e5d6 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:24.583877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1392/2793] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:59:24 2022] POST /identity/v3/projects => generated 367 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:59:24.588221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9915e8cd-ab0c-4c27-b794-11a0be242577 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:24.599489 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9915e8cd-ab0c-4c27-b794-11a0be242577 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:24.656602 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de73a64c-3832-45d3-a208-efa3c91dfd97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:24.657203 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de73a64c-3832-45d3-a208-efa3c91dfd97 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:24.657481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de73a64c-3832-45d3-a208-efa3c91dfd97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:24.657748 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de73a64c-3832-45d3-a208-efa3c91dfd97 admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:24.658392 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de73a64c-3832-45d3-a208-efa3c91dfd97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:24.660297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de73a64c-3832-45d3-a208-efa3c91dfd97 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:24.755859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1402/2794] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:59:24 2022] POST /identity/v3/users => generated 290 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:59:24.759894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74a6020-ab5c-45ac-9aa4-0678aa58e5d6 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:24.770363 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74a6020-ab5c-45ac-9aa4-0678aa58e5d6 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:24.775125 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72542a5-9008-4dc4-90d4-bae99a1022f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:24.775785 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72542a5-9008-4dc4-90d4-bae99a1022f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:24.783809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72542a5-9008-4dc4-90d4-bae99a1022f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:24.783809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72542a5-9008-4dc4-90d4-bae99a1022f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:24.783809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72542a5-9008-4dc4-90d4-bae99a1022f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:24.783809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72542a5-9008-4dc4-90d4-bae99a1022f2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:24.790500 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1393/2795] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:59:24.795306 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de73a64c-3832-45d3-a208-efa3c91dfd97 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:24.805257 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de73a64c-3832-45d3-a208-efa3c91dfd97 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:24.810583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1512a61b-591b-481c-a4d4-b9a9043261e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:24.811252 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1512a61b-591b-481c-a4d4-b9a9043261e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:24.811534 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1512a61b-591b-481c-a4d4-b9a9043261e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:24.811808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1512a61b-591b-481c-a4d4-b9a9043261e9 admin admin] PATH_INFO: `/v3/projects/b4c5d02a16fe492f830cfbfc74d214fc/users/38d77884d68741a0858a27e2ceb07569/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:24.826379 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1512a61b-591b-481c-a4d4-b9a9043261e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4c5d02a16fe492f830cfbfc74d214fc, user_id=38d77884d68741a0858a27e2ceb07569, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:24.829036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1512a61b-591b-481c-a4d4-b9a9043261e9 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:24.904938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1403/2796] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:59:24 2022] PUT /identity/v3/projects/b4c5d02a16fe492f830cfbfc74d214fc/users/38d77884d68741a0858a27e2ceb07569/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:24.909250 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72542a5-9008-4dc4-90d4-bae99a1022f2 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:24.914704 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:24.920967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72542a5-9008-4dc4-90d4-bae99a1022f2 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:24.981519 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e110148-94bf-439f-bdee-3e6147ac7cdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:24.982121 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e110148-94bf-439f-bdee-3e6147ac7cdf admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:24.982455 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e110148-94bf-439f-bdee-3e6147ac7cdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:24.982722 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e110148-94bf-439f-bdee-3e6147ac7cdf admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:24.983362 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e110148-94bf-439f-bdee-3e6147ac7cdf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:24.984938 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e110148-94bf-439f-bdee-3e6147ac7cdf admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:24.997249 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1394/2797] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:24 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:59:25.001236 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1512a61b-591b-481c-a4d4-b9a9043261e9 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:25.012069 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1512a61b-591b-481c-a4d4-b9a9043261e9 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:25.016720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce3476c-d11f-487d-876e-5d734ae81961 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:25.017409 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce3476c-d11f-487d-876e-5d734ae81961 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:25.017720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce3476c-d11f-487d-876e-5d734ae81961 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:25.017997 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce3476c-d11f-487d-876e-5d734ae81961 admin admin] PATH_INFO: `/v3/projects/b4c5d02a16fe492f830cfbfc74d214fc/users/38d77884d68741a0858a27e2ceb07569/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:25.019986 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce3476c-d11f-487d-876e-5d734ae81961 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4c5d02a16fe492f830cfbfc74d214fc, user_id=38d77884d68741a0858a27e2ceb07569, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:25.022845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce3476c-d11f-487d-876e-5d734ae81961 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:25.036503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1404/2798] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:59:25 2022] PUT /identity/v3/projects/b4c5d02a16fe492f830cfbfc74d214fc/users/38d77884d68741a0858a27e2ceb07569/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:25.047499 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66525283-3b2b-42c9-bd3e-76333ea15cbd None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:25.047782 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66525283-3b2b-42c9-bd3e-76333ea15cbd None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:25.048053 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66525283-3b2b-42c9-bd3e-76333ea15cbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:25.078125 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-66525283-3b2b-42c9-bd3e-76333ea15cbd None None] MFA Rules not processed for user `38d77884d68741a0858a27e2ceb07569`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:25.079393 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-66525283-3b2b-42c9-bd3e-76333ea15cbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:25.149478 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1395/2799] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:25 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:25.161879 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce3476c-d11f-487d-876e-5d734ae81961 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:25.163642 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6ce3476c-d11f-487d-876e-5d734ae81961 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:25.223129 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce3476c-d11f-487d-876e-5d734ae81961 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:25.227655 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98f5c00-241e-49e3-a35b-8677e157f5c1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:25.228290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98f5c00-241e-49e3-a35b-8677e157f5c1 service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:25.228538 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98f5c00-241e-49e3-a35b-8677e157f5c1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:25.229403 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98f5c00-241e-49e3-a35b-8677e157f5c1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:25.239901 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98f5c00-241e-49e3-a35b-8677e157f5c1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:25.243135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98f5c00-241e-49e3-a35b-8677e157f5c1 service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:25.255279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1405/2800] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:25 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:25.359040 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66525283-3b2b-42c9-bd3e-76333ea15cbd None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:25.368713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66525283-3b2b-42c9-bd3e-76333ea15cbd None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:25.372469 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc05fca1-577c-49ed-aadd-3477bddc285f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:25.374370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc05fca1-577c-49ed-aadd-3477bddc285f service cinder] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:25.374370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc05fca1-577c-49ed-aadd-3477bddc285f service cinder] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:25.374370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc05fca1-577c-49ed-aadd-3477bddc285f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:25.431088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc05fca1-577c-49ed-aadd-3477bddc285f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:25.432593 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc05fca1-577c-49ed-aadd-3477bddc285f service cinder] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:25.441570 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1396/2801] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:25 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:26.043686 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98f5c00-241e-49e3-a35b-8677e157f5c1 service cinder] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:26.046064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d98f5c00-241e-49e3-a35b-8677e157f5c1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:26.080948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:26.163262 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98f5c00-241e-49e3-a35b-8677e157f5c1 service cinder] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:26.170501 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd330255-9e31-4210-b325-6313ee635824 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:26.172499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd330255-9e31-4210-b325-6313ee635824 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:26.172499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd330255-9e31-4210-b325-6313ee635824 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:26.172499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd330255-9e31-4210-b325-6313ee635824 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:26.173875 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-dd330255-9e31-4210-b325-6313ee635824 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:26.188529 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd330255-9e31-4210-b325-6313ee635824 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:26.190275 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd330255-9e31-4210-b325-6313ee635824 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:26.202395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1406/2802] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:30.568733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc05fca1-577c-49ed-aadd-3477bddc285f service cinder] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:30.596314 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:30.634073 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc05fca1-577c-49ed-aadd-3477bddc285f service cinder] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:30.640015 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:30.640636 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:30.640915 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:30.641187 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] PATH_INFO: `/v3/users/b90bad4106dc4de99194a0a966932e58` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:30.642351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b90bad4106dc4de99194a0a966932e58)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:30.644335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:30.883267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b90bad4106dc4de99194a0a966932e58'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:30.883686 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b90bad4106dc4de99194a0a966932e58'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:30.891686 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b90bad4106dc4de99194a0a966932e58'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:30.904464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b90bad4106dc4de99194a0a966932e58'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:30.971134 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b90bad4106dc4de99194a0a966932e58'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:30.977365 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1397/2803] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:59:30 2022] DELETE /identity/v3/users/b90bad4106dc4de99194a0a966932e58 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:30.982484 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f1a9b6-dbfd-4bee-9aee-60f4c1045c26 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:30.982961 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f1a9b6-dbfd-4bee-9aee-60f4c1045c26 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:30.983070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f1a9b6-dbfd-4bee-9aee-60f4c1045c26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:31.007019 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-e9f1a9b6-dbfd-4bee-9aee-60f4c1045c26 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:31.012064 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e9f1a9b6-dbfd-4bee-9aee-60f4c1045c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:31.078708 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1407/2804] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:31.089175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:31.090616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:31.149672 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3902b9-21e4-49f4-9290-350b1ebc55b3 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:31.154055 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1240e057-445f-488b-83d7-9d9b6d53e828 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:31.154698 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1240e057-445f-488b-83d7-9d9b6d53e828 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:31.154974 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1240e057-445f-488b-83d7-9d9b6d53e828 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:31.155248 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1240e057-445f-488b-83d7-9d9b6d53e828 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:31.164796 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1240e057-445f-488b-83d7-9d9b6d53e828 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:31.166858 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1240e057-445f-488b-83d7-9d9b6d53e828 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:31.176991 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1398/2805] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:31.624816 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f1a9b6-dbfd-4bee-9aee-60f4c1045c26 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:31.626276 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-e9f1a9b6-dbfd-4bee-9aee-60f4c1045c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:31.639059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f1a9b6-dbfd-4bee-9aee-60f4c1045c26 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:31.643960 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:31.649053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:31.649053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:31.649053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] PATH_INFO: `/v3/projects/4dcd0d1ec0d3435e9b11b25df4200fc3` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:31.649053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4dcd0d1ec0d3435e9b11b25df4200fc3)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:31.649053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:31.780021 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4dcd0d1ec0d3435e9b11b25df4200fc3', 'request_id': 'req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:31.785717 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4dcd0d1ec0d3435e9b11b25df4200fc3', 'request_id': 'req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:31.908497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4dcd0d1ec0d3435e9b11b25df4200fc3', 'request_id': 'req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:31.920187 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] The token cache is being invalidate because project 4dcd0d1ec0d3435e9b11b25df4200fc3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:59:31.920474 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:31.921103 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:31.922635 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1408/2806] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:59:31 2022] DELETE /identity/v3/projects/4dcd0d1ec0d3435e9b11b25df4200fc3 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:31.926695 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1240e057-445f-488b-83d7-9d9b6d53e828 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:31.937563 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1240e057-445f-488b-83d7-9d9b6d53e828 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:31.979428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:31.993857 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:31.995072 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:31.995336 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:31.995604 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] PATH_INFO: `/v3/users/38d77884d68741a0858a27e2ceb07569` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:31.996735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=38d77884d68741a0858a27e2ceb07569)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:31.998689 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:32.208112 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38d77884d68741a0858a27e2ceb07569'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:32.208682 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38d77884d68741a0858a27e2ceb07569'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:32.215741 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38d77884d68741a0858a27e2ceb07569'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:32.227714 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38d77884d68741a0858a27e2ceb07569'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:32.302573 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38d77884d68741a0858a27e2ceb07569'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:32.309787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1399/2807] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:59:31 2022] DELETE /identity/v3/users/38d77884d68741a0858a27e2ceb07569 => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:32.882394 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:32.883952 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:32.899859 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd972f0-61b5-4f9e-89fd-e42da7ae9d85 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:32.915598 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:32.917269 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:32.942911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:32.972562 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95d98db-dd65-4873-b72f-c8b9e1081876 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:32.973190 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95d98db-dd65-4873-b72f-c8b9e1081876 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:32.973518 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95d98db-dd65-4873-b72f-c8b9e1081876 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:32.973726 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95d98db-dd65-4873-b72f-c8b9e1081876 admin admin] PATH_INFO: `/v3/projects/b4c5d02a16fe492f830cfbfc74d214fc` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:32.974851 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95d98db-dd65-4873-b72f-c8b9e1081876 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b4c5d02a16fe492f830cfbfc74d214fc)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:32.976982 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95d98db-dd65-4873-b72f-c8b9e1081876 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:32.987092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd431258-7ff8-4c99-aa9f-5f1076ed9e03 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:32.991764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22299fa9-d9dd-4e25-8b2e-ed3275e2f555 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:32.992444 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22299fa9-d9dd-4e25-8b2e-ed3275e2f555 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:32.992723 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22299fa9-d9dd-4e25-8b2e-ed3275e2f555 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:32.992998 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22299fa9-d9dd-4e25-8b2e-ed3275e2f555 admin admin] PATH_INFO: `/v3/users/fb84a5e40d78497aa1b8658b6962e349` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:32.994190 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22299fa9-d9dd-4e25-8b2e-ed3275e2f555 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb84a5e40d78497aa1b8658b6962e349)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:32.996269 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22299fa9-d9dd-4e25-8b2e-ed3275e2f555 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:33.122849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c95d98db-dd65-4873-b72f-c8b9e1081876 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b4c5d02a16fe492f830cfbfc74d214fc', 'request_id': 'req-c95d98db-dd65-4873-b72f-c8b9e1081876'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:33.130172 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c95d98db-dd65-4873-b72f-c8b9e1081876 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b4c5d02a16fe492f830cfbfc74d214fc', 'request_id': 'req-c95d98db-dd65-4873-b72f-c8b9e1081876'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:33.206070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c95d98db-dd65-4873-b72f-c8b9e1081876 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b4c5d02a16fe492f830cfbfc74d214fc', 'request_id': 'req-c95d98db-dd65-4873-b72f-c8b9e1081876'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:33.221722 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c95d98db-dd65-4873-b72f-c8b9e1081876 admin admin] The token cache is being invalidate because project b4c5d02a16fe492f830cfbfc74d214fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:59:33.222290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c95d98db-dd65-4873-b72f-c8b9e1081876 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:33.222712 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c95d98db-dd65-4873-b72f-c8b9e1081876 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:33.223958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-22299fa9-d9dd-4e25-8b2e-ed3275e2f555 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb84a5e40d78497aa1b8658b6962e349'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:33.224480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1409/2808] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:59:32 2022] DELETE /identity/v3/projects/b4c5d02a16fe492f830cfbfc74d214fc => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:33.224644 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-22299fa9-d9dd-4e25-8b2e-ed3275e2f555 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb84a5e40d78497aa1b8658b6962e349'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:33.231734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc1c655-b3d0-4d8e-8060-fba759dbd752 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:33.232406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc1c655-b3d0-4d8e-8060-fba759dbd752 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:33.235925 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc1c655-b3d0-4d8e-8060-fba759dbd752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:33.236088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-22299fa9-d9dd-4e25-8b2e-ed3275e2f555 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb84a5e40d78497aa1b8658b6962e349'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:33.241574 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=81,keystone:DELETE=18,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:33.247372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-22299fa9-d9dd-4e25-8b2e-ed3275e2f555 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb84a5e40d78497aa1b8658b6962e349'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:33.256426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-7dc1c655-b3d0-4d8e-8060-fba759dbd752 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:33.257657 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7dc1c655-b3d0-4d8e-8060-fba759dbd752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:33.321159 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1410/2809] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:59:33.331343 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c7f753-f974-4b8f-9df8-2637e42fda0b None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:33.331653 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c7f753-f974-4b8f-9df8-2637e42fda0b None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:33.331946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c7f753-f974-4b8f-9df8-2637e42fda0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:33.361956 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-27c7f753-f974-4b8f-9df8-2637e42fda0b None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:33.365352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-22299fa9-d9dd-4e25-8b2e-ed3275e2f555 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb84a5e40d78497aa1b8658b6962e349'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:33.365629 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-27c7f753-f974-4b8f-9df8-2637e42fda0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:33.373790 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1400/2810] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 03:59:32 2022] DELETE /identity/v3/users/fb84a5e40d78497aa1b8658b6962e349 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:59:33.375138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1411/2811] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:59:33.378308 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345cdff4-d82f-4552-b984-3f911cc704ec None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:33.378827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345cdff4-d82f-4552-b984-3f911cc704ec None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:33.379111 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345cdff4-d82f-4552-b984-3f911cc704ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:33.380025 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c7f753-f974-4b8f-9df8-2637e42fda0b None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:33.393399 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c7f753-f974-4b8f-9df8-2637e42fda0b None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:33.400649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-345cdff4-d82f-4552-b984-3f911cc704ec None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:33.410171 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-345cdff4-d82f-4552-b984-3f911cc704ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:33.461494 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:33.469305 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f42dff-49aa-42dd-83aa-d46facc58ba0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:33.469941 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f42dff-49aa-42dd-83aa-d46facc58ba0 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:33.473762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f42dff-49aa-42dd-83aa-d46facc58ba0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:33.473762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f42dff-49aa-42dd-83aa-d46facc58ba0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:33.474254 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1401/2812] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:33.474475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f42dff-49aa-42dd-83aa-d46facc58ba0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:59:33.474475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f42dff-49aa-42dd-83aa-d46facc58ba0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:33.474475 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f42dff-49aa-42dd-83aa-d46facc58ba0 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:33.482613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345cdff4-d82f-4552-b984-3f911cc704ec None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:33.484209 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-345cdff4-d82f-4552-b984-3f911cc704ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:33.485740 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1412/2813] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:59:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:59:33.491226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f42dff-49aa-42dd-83aa-d46facc58ba0 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:33.504043 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f42dff-49aa-42dd-83aa-d46facc58ba0 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:33.511045 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aedc8e6-27f1-4c22-99af-c83a14b94dea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:33.511866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aedc8e6-27f1-4c22-99af-c83a14b94dea admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:33.512367 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aedc8e6-27f1-4c22-99af-c83a14b94dea admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:33.512747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aedc8e6-27f1-4c22-99af-c83a14b94dea admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:33.513417 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aedc8e6-27f1-4c22-99af-c83a14b94dea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:33.515834 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aedc8e6-27f1-4c22-99af-c83a14b94dea admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:33.552911 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345cdff4-d82f-4552-b984-3f911cc704ec None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:33.557351 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd1f577-c584-4c0d-9e50-04ae4ea55bc5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:33.557987 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd1f577-c584-4c0d-9e50-04ae4ea55bc5 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:33.558318 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd1f577-c584-4c0d-9e50-04ae4ea55bc5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:33.558616 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd1f577-c584-4c0d-9e50-04ae4ea55bc5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:33.568267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd1f577-c584-4c0d-9e50-04ae4ea55bc5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:33.569798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd1f577-c584-4c0d-9e50-04ae4ea55bc5 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:33.580775 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1402/2814] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:59:33.623277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1413/2815] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:59:33 2022] POST /identity/v3/projects => generated 365 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:59:33.624204 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd1f577-c584-4c0d-9e50-04ae4ea55bc5 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:33.634264 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd1f577-c584-4c0d-9e50-04ae4ea55bc5 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:33.697379 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b816d40-3a2b-41d2-9120-c662af6c44c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:33.698390 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b816d40-3a2b-41d2-9120-c662af6c44c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:33.698675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b816d40-3a2b-41d2-9120-c662af6c44c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:33.699011 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b816d40-3a2b-41d2-9120-c662af6c44c1 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:33.699622 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b816d40-3a2b-41d2-9120-c662af6c44c1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:33.701743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b816d40-3a2b-41d2-9120-c662af6c44c1 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:33.806358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1403/2816] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:59:33 2022] POST /identity/v3/users => generated 283 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 03:59:33.811435 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aedc8e6-27f1-4c22-99af-c83a14b94dea admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:33.821918 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aedc8e6-27f1-4c22-99af-c83a14b94dea admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:33.829372 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fe3750-c3b2-474e-a6ab-fda800297c00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:33.830440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe3750-c3b2-474e-a6ab-fda800297c00 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:33.830938 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe3750-c3b2-474e-a6ab-fda800297c00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:33.831408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe3750-c3b2-474e-a6ab-fda800297c00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:33.834173 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fe3750-c3b2-474e-a6ab-fda800297c00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:33.835844 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fe3750-c3b2-474e-a6ab-fda800297c00 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:33.854290 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1414/2817] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:33 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:59:33.855740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b816d40-3a2b-41d2-9120-c662af6c44c1 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:33.866864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b816d40-3a2b-41d2-9120-c662af6c44c1 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:33.873703 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184b9a3e-74d1-4c24-b443-a386e767f07b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:33.874429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184b9a3e-74d1-4c24-b443-a386e767f07b admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:33.874781 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184b9a3e-74d1-4c24-b443-a386e767f07b admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:33.875123 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184b9a3e-74d1-4c24-b443-a386e767f07b admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:33.876088 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184b9a3e-74d1-4c24-b443-a386e767f07b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:33.877928 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184b9a3e-74d1-4c24-b443-a386e767f07b admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:33.890388 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1404/2818] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:33 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:59:33.894843 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fe3750-c3b2-474e-a6ab-fda800297c00 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:33.905358 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fe3750-c3b2-474e-a6ab-fda800297c00 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:33.910408 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e70156-557b-4ac5-ba61-db76097f03f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:33.914297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e70156-557b-4ac5-ba61-db76097f03f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:33.914297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e70156-557b-4ac5-ba61-db76097f03f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:33.914297 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e70156-557b-4ac5-ba61-db76097f03f7 admin admin] PATH_INFO: `/v3/projects/65a57c5d95854108a82749461d58d68e/users/ef903554c5ad4e6aa3b64a1936099afd/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:33.927710 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e70156-557b-4ac5-ba61-db76097f03f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=65a57c5d95854108a82749461d58d68e, user_id=ef903554c5ad4e6aa3b64a1936099afd, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:33.930537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e70156-557b-4ac5-ba61-db76097f03f7 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:34.005833 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1415/2819] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:59:33 2022] PUT /identity/v3/projects/65a57c5d95854108a82749461d58d68e/users/ef903554c5ad4e6aa3b64a1936099afd/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:59:34.059205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184b9a3e-74d1-4c24-b443-a386e767f07b admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:34.064544 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-184b9a3e-74d1-4c24-b443-a386e767f07b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:34.071896 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:34.074631 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfda04bd-6ef8-40d3-9484-60e2c1700b88 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:34.075090 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfda04bd-6ef8-40d3-9484-60e2c1700b88 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:34.075371 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfda04bd-6ef8-40d3-9484-60e2c1700b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:34.081845 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184b9a3e-74d1-4c24-b443-a386e767f07b admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:34.106174 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-bfda04bd-6ef8-40d3-9484-60e2c1700b88 None None] MFA Rules not processed for user `ef903554c5ad4e6aa3b64a1936099afd`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:34.107450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-bfda04bd-6ef8-40d3-9484-60e2c1700b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:34.152806 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a70ae2a-266a-4327-ac43-8a6849ad79de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:34.153605 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a70ae2a-266a-4327-ac43-8a6849ad79de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:34.153886 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a70ae2a-266a-4327-ac43-8a6849ad79de admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:34.154197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a70ae2a-266a-4327-ac43-8a6849ad79de admin admin] PATH_INFO: `/v3/projects/bd69fc15eb864544a327441cc196eddc` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:34.155399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a70ae2a-266a-4327-ac43-8a6849ad79de admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd69fc15eb864544a327441cc196eddc)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:34.157372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a70ae2a-266a-4327-ac43-8a6849ad79de admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:34.178197 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1416/2820] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:34 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 03:59:34.188582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfda04bd-6ef8-40d3-9484-60e2c1700b88 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:34.189976 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-bfda04bd-6ef8-40d3-9484-60e2c1700b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:34.236634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-2a70ae2a-266a-4327-ac43-8a6849ad79de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd69fc15eb864544a327441cc196eddc', 'request_id': 'req-2a70ae2a-266a-4327-ac43-8a6849ad79de'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:34.242210 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-2a70ae2a-266a-4327-ac43-8a6849ad79de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd69fc15eb864544a327441cc196eddc', 'request_id': 'req-2a70ae2a-266a-4327-ac43-8a6849ad79de'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:34.248609 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-2a70ae2a-266a-4327-ac43-8a6849ad79de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd69fc15eb864544a327441cc196eddc', 'request_id': 'req-2a70ae2a-266a-4327-ac43-8a6849ad79de'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:34.248910 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfda04bd-6ef8-40d3-9484-60e2c1700b88 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:34.283137 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:34.300469 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb661ee-8022-4d65-9f4c-ac75d1a28c7b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:34.301114 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb661ee-8022-4d65-9f4c-ac75d1a28c7b service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:34.301426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb661ee-8022-4d65-9f4c-ac75d1a28c7b service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:34.301705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb661ee-8022-4d65-9f4c-ac75d1a28c7b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:34.380791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb661ee-8022-4d65-9f4c-ac75d1a28c7b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:34.382913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb661ee-8022-4d65-9f4c-ac75d1a28c7b service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:34.395196 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1417/2821] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:34 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 03:59:34.408803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-2a70ae2a-266a-4327-ac43-8a6849ad79de admin admin] The token cache is being invalidate because project bd69fc15eb864544a327441cc196eddc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 03:59:34.408803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-2a70ae2a-266a-4327-ac43-8a6849ad79de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:34.408803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-2a70ae2a-266a-4327-ac43-8a6849ad79de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 03:59:34.408803 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1405/2822] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 03:59:34 2022] DELETE /identity/v3/projects/bd69fc15eb864544a327441cc196eddc => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 03:59:34.420423 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9527d063-8e03-44a9-90c8-02d0cd7ea8c3 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:34.420707 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9527d063-8e03-44a9-90c8-02d0cd7ea8c3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:34.421100 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9527d063-8e03-44a9-90c8-02d0cd7ea8c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:34.442217 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-9527d063-8e03-44a9-90c8-02d0cd7ea8c3 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:34.443497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-9527d063-8e03-44a9-90c8-02d0cd7ea8c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:34.505152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1418/2823] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:34.513139 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b419c-6996-45eb-8ad7-5c7134fa6843 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:34.513431 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b419c-6996-45eb-8ad7-5c7134fa6843 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:34.513704 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b419c-6996-45eb-8ad7-5c7134fa6843 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:34.536238 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-ef7b419c-6996-45eb-8ad7-5c7134fa6843 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:34.537380 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-ef7b419c-6996-45eb-8ad7-5c7134fa6843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:34.546205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1406/2824] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:34.550428 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9527d063-8e03-44a9-90c8-02d0cd7ea8c3 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:34.560946 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9527d063-8e03-44a9-90c8-02d0cd7ea8c3 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:34.565582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7318b4-ff8c-4e8c-81e4-29a8f411bcef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:34.569153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7318b4-ff8c-4e8c-81e4-29a8f411bcef admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:34.569153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7318b4-ff8c-4e8c-81e4-29a8f411bcef admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:34.569153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7318b4-ff8c-4e8c-81e4-29a8f411bcef admin admin] PATH_INFO: `/v3/domains` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:34.569153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7318b4-ff8c-4e8c-81e4-29a8f411bcef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81657) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 03:59:34.569153 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7318b4-ff8c-4e8c-81e4-29a8f411bcef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:34.569745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7318b4-ff8c-4e8c-81e4-29a8f411bcef admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:34.581054 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1419/2825] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 03:59:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 03:59:34.585366 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7b419c-6996-45eb-8ad7-5c7134fa6843 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:34.594796 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7b419c-6996-45eb-8ad7-5c7134fa6843 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:34.599511 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a72d45-77c0-440e-99de-4369e493d7d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:34.606004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a72d45-77c0-440e-99de-4369e493d7d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:34.606004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a72d45-77c0-440e-99de-4369e493d7d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:34.606004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a72d45-77c0-440e-99de-4369e493d7d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:34.606004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a72d45-77c0-440e-99de-4369e493d7d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:34.606004 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a72d45-77c0-440e-99de-4369e493d7d2 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:34.692708 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1407/2826] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 03:59:34 2022] POST /identity/v3/projects => generated 369 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:59:34.697046 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7318b4-ff8c-4e8c-81e4-29a8f411bcef admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:34.708185 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7318b4-ff8c-4e8c-81e4-29a8f411bcef admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:34.766791 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3521354-a2fd-497e-887a-fce733c7ffcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:34.767307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3521354-a2fd-497e-887a-fce733c7ffcc admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:34.767592 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3521354-a2fd-497e-887a-fce733c7ffcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:34.767862 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3521354-a2fd-497e-887a-fce733c7ffcc admin admin] PATH_INFO: `/v3/users` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:34.768498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3521354-a2fd-497e-887a-fce733c7ffcc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:34.770427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3521354-a2fd-497e-887a-fce733c7ffcc admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:34.858936 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1420/2827] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 03:59:34 2022] POST /identity/v3/users => generated 285 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 03:59:34.863466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a72d45-77c0-440e-99de-4369e493d7d2 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:34.878621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a72d45-77c0-440e-99de-4369e493d7d2 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:34.884622 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38421ca2-a592-48c1-b3dc-da3aecd2a9c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:34.885311 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38421ca2-a592-48c1-b3dc-da3aecd2a9c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:34.885664 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38421ca2-a592-48c1-b3dc-da3aecd2a9c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:34.886021 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38421ca2-a592-48c1-b3dc-da3aecd2a9c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:34.887528 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38421ca2-a592-48c1-b3dc-da3aecd2a9c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:34.890545 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38421ca2-a592-48c1-b3dc-da3aecd2a9c1 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:34.902464 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1408/2828] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:34 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 03:59:34.907034 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3521354-a2fd-497e-887a-fce733c7ffcc admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:34.918272 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3521354-a2fd-497e-887a-fce733c7ffcc admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:34.923186 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1ace19-8473-4d9c-a2d4-cd178071d848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:34.923900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1ace19-8473-4d9c-a2d4-cd178071d848 admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:34.924228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1ace19-8473-4d9c-a2d4-cd178071d848 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:34.924506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1ace19-8473-4d9c-a2d4-cd178071d848 admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:34.925156 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1ace19-8473-4d9c-a2d4-cd178071d848 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:34.926906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1ace19-8473-4d9c-a2d4-cd178071d848 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:34.929993 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:34.942206 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1421/2829] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 03:59:34 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 03:59:34.943505 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38421ca2-a592-48c1-b3dc-da3aecd2a9c1 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:34.954542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38421ca2-a592-48c1-b3dc-da3aecd2a9c1 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:34.959412 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1487b60d-12c8-4ded-b5f4-75897b3fb535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:34.960061 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1487b60d-12c8-4ded-b5f4-75897b3fb535 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:34.960337 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1487b60d-12c8-4ded-b5f4-75897b3fb535 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:34.960608 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1487b60d-12c8-4ded-b5f4-75897b3fb535 admin admin] PATH_INFO: `/v3/projects/fef5284c8ed64126af0a060258f63161/users/9862639292074df2859d0be5c283cc5f/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:34.976970 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1487b60d-12c8-4ded-b5f4-75897b3fb535 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fef5284c8ed64126af0a060258f63161, user_id=9862639292074df2859d0be5c283cc5f, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:34.980137 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1487b60d-12c8-4ded-b5f4-75897b3fb535 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:35.077467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1409/2830] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 03:59:34 2022] PUT /identity/v3/projects/fef5284c8ed64126af0a060258f63161/users/9862639292074df2859d0be5c283cc5f/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 03:59:35.096852 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bedf4ce-16ae-42e4-84f2-2604a6af66ba None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:35.097416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bedf4ce-16ae-42e4-84f2-2604a6af66ba None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:35.097416 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bedf4ce-16ae-42e4-84f2-2604a6af66ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:35.151947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5bedf4ce-16ae-42e4-84f2-2604a6af66ba None None] MFA Rules not processed for user `9862639292074df2859d0be5c283cc5f`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 03:59:35.151947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5bedf4ce-16ae-42e4-84f2-2604a6af66ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:35.205499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1422/2831] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 03:59:35 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 03:59:35.222417 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1487b60d-12c8-4ded-b5f4-75897b3fb535 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:35.224252 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-1487b60d-12c8-4ded-b5f4-75897b3fb535 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:35.262958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:35.311986 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1487b60d-12c8-4ded-b5f4-75897b3fb535 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:35.316127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad690b7-e208-421a-9beb-82826e93dcd5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f3b3c7ebc08496e970f8f25fd6395e4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:35.316832 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad690b7-e208-421a-9beb-82826e93dcd5 service glance] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:35.317106 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad690b7-e208-421a-9beb-82826e93dcd5 service glance] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:35.317370 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad690b7-e208-421a-9beb-82826e93dcd5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:35.326425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad690b7-e208-421a-9beb-82826e93dcd5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:35.327844 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad690b7-e208-421a-9beb-82826e93dcd5 service glance] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:35.337740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1410/2832] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:35 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:41.055105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bedf4ce-16ae-42e4-84f2-2604a6af66ba None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 03:59:41.056654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5bedf4ce-16ae-42e4-84f2-2604a6af66ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:41.131906 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bedf4ce-16ae-42e4-84f2-2604a6af66ba None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 03:59:41.135522 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51622381-c583-4162-a675-e6e0bb954948 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 03:59:41.136177 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51622381-c583-4162-a675-e6e0bb954948 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 03:59:41.136405 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51622381-c583-4162-a675-e6e0bb954948 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 03:59:41.136722 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51622381-c583-4162-a675-e6e0bb954948 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 03:59:41.138745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-51622381-c583-4162-a675-e6e0bb954948 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 03:59:41.210616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51622381-c583-4162-a675-e6e0bb954948 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 03:59:41.210616 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51622381-c583-4162-a675-e6e0bb954948 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 03:59:41.222266 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1423/2833] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 03:59:41 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 03:59:45.554653 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=21 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 03:59:51.216310 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=83 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:11.062787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad690b7-e208-421a-9beb-82826e93dcd5 service glance] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:11.124281 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad690b7-e208-421a-9beb-82826e93dcd5 service glance] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:11.129160 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:11.129808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:11.130101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:11.130407 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] PATH_INFO: `/v3/users/9862639292074df2859d0be5c283cc5f` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:11.131548 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9862639292074df2859d0be5c283cc5f)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:11.133507 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:11.345888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9862639292074df2859d0be5c283cc5f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:11.346628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9862639292074df2859d0be5c283cc5f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:11.353924 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9862639292074df2859d0be5c283cc5f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:11.366305 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9862639292074df2859d0be5c283cc5f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:11.439459 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9862639292074df2859d0be5c283cc5f'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:11.445958 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1411/2834] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 04:00:11 2022] DELETE /identity/v3/users/9862639292074df2859d0be5c283cc5f => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:00:11.450528 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a06cbc-968c-4800-b70d-01ed1483e637 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:11.451004 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a06cbc-968c-4800-b70d-01ed1483e637 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:11.451117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a06cbc-968c-4800-b70d-01ed1483e637 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:11.471213 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-a2a06cbc-968c-4800-b70d-01ed1483e637 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:11.472486 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a2a06cbc-968c-4800-b70d-01ed1483e637 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:11.530505 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1424/2835] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 04:00:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 04:00:11.540954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:11.542481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:11.599345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d6fdaf-c9d1-42b2-8c12-79d0e1cd5312 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:11.603235 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf848d2b-345c-4c6b-b2ee-f5b8be1f220c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:11.603829 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf848d2b-345c-4c6b-b2ee-f5b8be1f220c service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:11.604108 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf848d2b-345c-4c6b-b2ee-f5b8be1f220c service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:11.604377 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf848d2b-345c-4c6b-b2ee-f5b8be1f220c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:11.613397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf848d2b-345c-4c6b-b2ee-f5b8be1f220c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:11.614859 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf848d2b-345c-4c6b-b2ee-f5b8be1f220c service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:11.624217 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1412/2836] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:00:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:00:12.024495 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a06cbc-968c-4800-b70d-01ed1483e637 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:12.028027 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a2a06cbc-968c-4800-b70d-01ed1483e637 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:12.039556 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a06cbc-968c-4800-b70d-01ed1483e637 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:12.044216 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02791aca-1d8d-4f01-89a3-02cee061d348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:12.045296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02791aca-1d8d-4f01-89a3-02cee061d348 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:12.045296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02791aca-1d8d-4f01-89a3-02cee061d348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:12.045579 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02791aca-1d8d-4f01-89a3-02cee061d348 admin admin] PATH_INFO: `/v3/projects/fef5284c8ed64126af0a060258f63161` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:12.046583 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02791aca-1d8d-4f01-89a3-02cee061d348 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fef5284c8ed64126af0a060258f63161)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:12.048691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02791aca-1d8d-4f01-89a3-02cee061d348 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:12.142811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-02791aca-1d8d-4f01-89a3-02cee061d348 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fef5284c8ed64126af0a060258f63161', 'request_id': 'req-02791aca-1d8d-4f01-89a3-02cee061d348'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:12.148279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-02791aca-1d8d-4f01-89a3-02cee061d348 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fef5284c8ed64126af0a060258f63161', 'request_id': 'req-02791aca-1d8d-4f01-89a3-02cee061d348'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:12.216285 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-02791aca-1d8d-4f01-89a3-02cee061d348 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fef5284c8ed64126af0a060258f63161', 'request_id': 'req-02791aca-1d8d-4f01-89a3-02cee061d348'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:12.228450 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-02791aca-1d8d-4f01-89a3-02cee061d348 admin admin] The token cache is being invalidate because project fef5284c8ed64126af0a060258f63161 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 04:00:12.230363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-02791aca-1d8d-4f01-89a3-02cee061d348 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:12.230363 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-02791aca-1d8d-4f01-89a3-02cee061d348 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:12.230919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1425/2837] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 04:00:12 2022] DELETE /identity/v3/projects/fef5284c8ed64126af0a060258f63161 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:00:12.236352 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0142b20-9d02-412b-a7ea-1c369073ee3a None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:12.236925 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0142b20-9d02-412b-a7ea-1c369073ee3a None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:12.237080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0142b20-9d02-412b-a7ea-1c369073ee3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:12.257383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-a0142b20-9d02-412b-a7ea-1c369073ee3a None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:12.258625 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a0142b20-9d02-412b-a7ea-1c369073ee3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:12.276059 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:12.319224 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1413/2838] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 04:00:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 04:00:12.327138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952e10e3-fbd6-4f2b-b940-198d5c9a9e4c None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:12.327430 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952e10e3-fbd6-4f2b-b940-198d5c9a9e4c None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:12.327735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952e10e3-fbd6-4f2b-b940-198d5c9a9e4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:12.347588 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-952e10e3-fbd6-4f2b-b940-198d5c9a9e4c None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:12.348811 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-952e10e3-fbd6-4f2b-b940-198d5c9a9e4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:12.357093 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1426/2839] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 04:00:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 04:00:12.361851 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0142b20-9d02-412b-a7ea-1c369073ee3a None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:12.371713 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0142b20-9d02-412b-a7ea-1c369073ee3a None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:12.377071 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80efb81e-78bb-44e1-a192-26955717a8a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:12.379534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80efb81e-78bb-44e1-a192-26955717a8a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:12.379534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80efb81e-78bb-44e1-a192-26955717a8a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:12.379534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80efb81e-78bb-44e1-a192-26955717a8a7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:12.379534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80efb81e-78bb-44e1-a192-26955717a8a7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 04:00:12.379534 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80efb81e-78bb-44e1-a192-26955717a8a7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:12.380767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80efb81e-78bb-44e1-a192-26955717a8a7 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:12.392893 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1414/2840] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 04:00:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 24 04:00:12.397274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952e10e3-fbd6-4f2b-b940-198d5c9a9e4c None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:12.407347 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952e10e3-fbd6-4f2b-b940-198d5c9a9e4c None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:12.412395 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a85277-44c2-497e-a990-b544841c9eed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:12.413086 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a85277-44c2-497e-a990-b544841c9eed admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:12.414139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a85277-44c2-497e-a990-b544841c9eed admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:12.414139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a85277-44c2-497e-a990-b544841c9eed admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:12.414448 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a85277-44c2-497e-a990-b544841c9eed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:12.416351 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a85277-44c2-497e-a990-b544841c9eed admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:12.538797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1427/2841] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 04:00:12 2022] POST /identity/v3/projects => generated 371 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 04:00:12.543070 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80efb81e-78bb-44e1-a192-26955717a8a7 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:12.553767 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80efb81e-78bb-44e1-a192-26955717a8a7 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:12.611345 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f130cc3-a9f3-4a52-a9ab-a42944bcf5eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:12.611944 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f130cc3-a9f3-4a52-a9ab-a42944bcf5eb admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:12.612219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f130cc3-a9f3-4a52-a9ab-a42944bcf5eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:12.612488 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f130cc3-a9f3-4a52-a9ab-a42944bcf5eb admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:12.613100 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f130cc3-a9f3-4a52-a9ab-a42944bcf5eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:12.615030 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f130cc3-a9f3-4a52-a9ab-a42944bcf5eb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:12.731097 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1415/2842] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 04:00:12 2022] POST /identity/v3/users => generated 286 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 04:00:12.735296 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a85277-44c2-497e-a990-b544841c9eed admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:12.746866 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a85277-44c2-497e-a990-b544841c9eed admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:12.751869 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aed1a17-6b42-4fe2-b751-0ad17843bb9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:12.752463 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aed1a17-6b42-4fe2-b751-0ad17843bb9f admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:12.752742 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aed1a17-6b42-4fe2-b751-0ad17843bb9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:12.753011 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aed1a17-6b42-4fe2-b751-0ad17843bb9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:12.753647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aed1a17-6b42-4fe2-b751-0ad17843bb9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:12.755267 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aed1a17-6b42-4fe2-b751-0ad17843bb9f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:12.765005 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1428/2843] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 04:00:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 04:00:12.769201 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f130cc3-a9f3-4a52-a9ab-a42944bcf5eb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:12.779507 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f130cc3-a9f3-4a52-a9ab-a42944bcf5eb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:12.784865 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ff651b-6dc9-408d-a139-4618ed1447d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:12.785462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ff651b-6dc9-408d-a139-4618ed1447d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:12.785743 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ff651b-6dc9-408d-a139-4618ed1447d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:12.786017 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ff651b-6dc9-408d-a139-4618ed1447d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:12.786715 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ff651b-6dc9-408d-a139-4618ed1447d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:12.788404 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ff651b-6dc9-408d-a139-4618ed1447d0 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:12.799816 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1416/2844] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 04:00:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 04:00:12.803985 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aed1a17-6b42-4fe2-b751-0ad17843bb9f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:12.814012 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aed1a17-6b42-4fe2-b751-0ad17843bb9f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:12.818855 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec1b788-4335-47eb-8b39-207350c0cada admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:12.819497 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec1b788-4335-47eb-8b39-207350c0cada admin admin] REQUEST_METHOD: `PUT` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:12.819779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec1b788-4335-47eb-8b39-207350c0cada admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:12.820070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec1b788-4335-47eb-8b39-207350c0cada admin admin] PATH_INFO: `/v3/projects/640210fe7e55404c9bdc0dcbadc7252c/users/a727f0ee89554bff9ba51633c53d8c47/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:12.835038 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec1b788-4335-47eb-8b39-207350c0cada admin admin] RBAC: Authorizing `identity:create_grant(project_id=640210fe7e55404c9bdc0dcbadc7252c, user_id=a727f0ee89554bff9ba51633c53d8c47, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:12.837687 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec1b788-4335-47eb-8b39-207350c0cada admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:12.897429 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1429/2845] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 04:00:12 2022] PUT /identity/v3/projects/640210fe7e55404c9bdc0dcbadc7252c/users/a727f0ee89554bff9ba51633c53d8c47/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:00:12.908696 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b524dc-d9b1-4f24-8d29-76f25871af79 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:12.908980 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b524dc-d9b1-4f24-8d29-76f25871af79 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:12.909252 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b524dc-d9b1-4f24-8d29-76f25871af79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:12.937202 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-40b524dc-d9b1-4f24-8d29-76f25871af79 None None] MFA Rules not processed for user `a727f0ee89554bff9ba51633c53d8c47`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:12.938481 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-40b524dc-d9b1-4f24-8d29-76f25871af79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:13.003302 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1417/2846] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 04:00:12 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 04:00:13.014464 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec1b788-4335-47eb-8b39-207350c0cada admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:13.016166 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1ec1b788-4335-47eb-8b39-207350c0cada admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:13.039847 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:13.076049 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec1b788-4335-47eb-8b39-207350c0cada admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:13.080364 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9bdbb2-146a-4e6c-9bcb-6ca59962347c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbfbfd64573141f88040692119ac5a1f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:13.080979 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9bdbb2-146a-4e6c-9bcb-6ca59962347c service cinder] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:13.081260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9bdbb2-146a-4e6c-9bcb-6ca59962347c service cinder] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:13.081535 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9bdbb2-146a-4e6c-9bcb-6ca59962347c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:13.091426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9bdbb2-146a-4e6c-9bcb-6ca59962347c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:13.092860 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9bdbb2-146a-4e6c-9bcb-6ca59962347c service cinder] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:13.103734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1430/2847] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:00:13 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:00:22.920074 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b524dc-d9b1-4f24-8d29-76f25871af79 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:22.921483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-40b524dc-d9b1-4f24-8d29-76f25871af79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:22.942441 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:22.986888 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b524dc-d9b1-4f24-8d29-76f25871af79 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:22.992265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:22.992899 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:22.993178 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:22.993450 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] PATH_INFO: `/v3/users/ef903554c5ad4e6aa3b64a1936099afd` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:22.994596 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef903554c5ad4e6aa3b64a1936099afd)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:22.996507 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:23.196150 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef903554c5ad4e6aa3b64a1936099afd'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:23.196800 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef903554c5ad4e6aa3b64a1936099afd'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:23.203755 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef903554c5ad4e6aa3b64a1936099afd'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:23.210447 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=17 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:23.216740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef903554c5ad4e6aa3b64a1936099afd'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:23.296428 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef903554c5ad4e6aa3b64a1936099afd'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:23.303335 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1418/2848] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 04:00:22 2022] DELETE /identity/v3/users/ef903554c5ad4e6aa3b64a1936099afd => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:00:23.307842 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dffe6d8-af25-47aa-8b1c-158ba70de4d5 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:23.308124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dffe6d8-af25-47aa-8b1c-158ba70de4d5 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:23.314343 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dffe6d8-af25-47aa-8b1c-158ba70de4d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:23.328228 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-5dffe6d8-af25-47aa-8b1c-158ba70de4d5 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:23.329537 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5dffe6d8-af25-47aa-8b1c-158ba70de4d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:23.388658 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1431/2849] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 04:00:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 04:00:23.399275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:23.406225 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:23.471349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8938e332-62fe-495a-871a-b2482b5cd01d admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:23.471349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a92287-9a28-421a-84b9-a38ae8876889 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:23.471349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a92287-9a28-421a-84b9-a38ae8876889 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:23.471349 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a92287-9a28-421a-84b9-a38ae8876889 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:23.471876 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a92287-9a28-421a-84b9-a38ae8876889 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:23.480908 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a92287-9a28-421a-84b9-a38ae8876889 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:23.482372 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a92287-9a28-421a-84b9-a38ae8876889 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:23.492086 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1419/2850] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:00:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:00:23.920419 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dffe6d8-af25-47aa-8b1c-158ba70de4d5 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:23.921861 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5dffe6d8-af25-47aa-8b1c-158ba70de4d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:23.934539 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dffe6d8-af25-47aa-8b1c-158ba70de4d5 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:23.939594 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efe7e8f-94ab-485a-b4e6-f2540039cc12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:23.940294 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efe7e8f-94ab-485a-b4e6-f2540039cc12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:23.940575 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efe7e8f-94ab-485a-b4e6-f2540039cc12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:23.940849 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efe7e8f-94ab-485a-b4e6-f2540039cc12 admin admin] PATH_INFO: `/v3/projects/65a57c5d95854108a82749461d58d68e` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:23.942135 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efe7e8f-94ab-485a-b4e6-f2540039cc12 admin admin] RBAC: Authorizing `identity:delete_project(project_id=65a57c5d95854108a82749461d58d68e)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:23.945561 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efe7e8f-94ab-485a-b4e6-f2540039cc12 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:24.051152 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8efe7e8f-94ab-485a-b4e6-f2540039cc12 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65a57c5d95854108a82749461d58d68e', 'request_id': 'req-8efe7e8f-94ab-485a-b4e6-f2540039cc12'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:24.057292 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8efe7e8f-94ab-485a-b4e6-f2540039cc12 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65a57c5d95854108a82749461d58d68e', 'request_id': 'req-8efe7e8f-94ab-485a-b4e6-f2540039cc12'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:24.119107 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8efe7e8f-94ab-485a-b4e6-f2540039cc12 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65a57c5d95854108a82749461d58d68e', 'request_id': 'req-8efe7e8f-94ab-485a-b4e6-f2540039cc12'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:24.130803 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8efe7e8f-94ab-485a-b4e6-f2540039cc12 admin admin] The token cache is being invalidate because project 65a57c5d95854108a82749461d58d68e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 04:00:24.131260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8efe7e8f-94ab-485a-b4e6-f2540039cc12 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:24.132783 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8efe7e8f-94ab-485a-b4e6-f2540039cc12 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:24.134444 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1432/2851] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 04:00:23 2022] DELETE /identity/v3/projects/65a57c5d95854108a82749461d58d68e => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:00:24.139440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a473be-a48c-4f3e-a1e0-4a2dfefd578d None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:24.139919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a473be-a48c-4f3e-a1e0-4a2dfefd578d None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:24.140040 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a473be-a48c-4f3e-a1e0-4a2dfefd578d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:24.160717 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-d7a473be-a48c-4f3e-a1e0-4a2dfefd578d None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:24.161959 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d7a473be-a48c-4f3e-a1e0-4a2dfefd578d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:24.198231 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:24.223024 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1420/2852] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 04:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 04:00:24.231451 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c638bc3a-e3ec-43c5-99f5-d607c9c9650f None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:24.231963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c638bc3a-e3ec-43c5-99f5-d607c9c9650f None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:24.232096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c638bc3a-e3ec-43c5-99f5-d607c9c9650f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:24.252913 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-c638bc3a-e3ec-43c5-99f5-d607c9c9650f None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:24.257277 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-c638bc3a-e3ec-43c5-99f5-d607c9c9650f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:24.263081 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1433/2853] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 04:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 04:00:24.267460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a473be-a48c-4f3e-a1e0-4a2dfefd578d None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:24.277109 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a473be-a48c-4f3e-a1e0-4a2dfefd578d None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:24.281764 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6484a21a-b3db-4168-81da-f8f72ab4edde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:24.282397 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6484a21a-b3db-4168-81da-f8f72ab4edde admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:24.282674 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6484a21a-b3db-4168-81da-f8f72ab4edde admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:24.282942 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6484a21a-b3db-4168-81da-f8f72ab4edde admin admin] PATH_INFO: `/v3/domains` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:24.283579 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6484a21a-b3db-4168-81da-f8f72ab4edde admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81658) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 24 04:00:24.283875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6484a21a-b3db-4168-81da-f8f72ab4edde admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:24.285440 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6484a21a-b3db-4168-81da-f8f72ab4edde admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:24.297462 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1421/2854] 10.222.0.65 () {58 vars in 1153 bytes} [Wed Aug 24 04:00:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 24 04:00:24.301856 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c638bc3a-e3ec-43c5-99f5-d607c9c9650f None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:24.311877 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c638bc3a-e3ec-43c5-99f5-d607c9c9650f None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:24.316639 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e133c0d-ff1d-4bfa-a77b-d2b26d76a529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:24.317223 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e133c0d-ff1d-4bfa-a77b-d2b26d76a529 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:24.317498 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e133c0d-ff1d-4bfa-a77b-d2b26d76a529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:24.317762 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e133c0d-ff1d-4bfa-a77b-d2b26d76a529 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:24.318432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e133c0d-ff1d-4bfa-a77b-d2b26d76a529 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:24.321268 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e133c0d-ff1d-4bfa-a77b-d2b26d76a529 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:24.388704 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1434/2855] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 04:00:24 2022] POST /identity/v3/projects => generated 385 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 04:00:24.392831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6484a21a-b3db-4168-81da-f8f72ab4edde admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:24.403069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6484a21a-b3db-4168-81da-f8f72ab4edde admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:24.461323 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fec7789-47ab-4f65-adc6-b9199ced2ebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:24.461827 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fec7789-47ab-4f65-adc6-b9199ced2ebb admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:24.462102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fec7789-47ab-4f65-adc6-b9199ced2ebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:24.462467 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fec7789-47ab-4f65-adc6-b9199ced2ebb admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:24.463097 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fec7789-47ab-4f65-adc6-b9199ced2ebb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:24.465029 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fec7789-47ab-4f65-adc6-b9199ced2ebb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:24.479900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e133c0d-ff1d-4bfa-a77b-d2b26d76a529 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:24.481638 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-7e133c0d-ff1d-4bfa-a77b-d2b26d76a529 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:24.495076 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e133c0d-ff1d-4bfa-a77b-d2b26d76a529 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:24.499735 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65e5b33-d458-45d2-8cfb-ae6695658087 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:24.500389 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e5b33-d458-45d2-8cfb-ae6695658087 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:24.500673 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e5b33-d458-45d2-8cfb-ae6695658087 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:24.500948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e5b33-d458-45d2-8cfb-ae6695658087 admin admin] PATH_INFO: `/v3/users/a727f0ee89554bff9ba51633c53d8c47` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:24.502085 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65e5b33-d458-45d2-8cfb-ae6695658087 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a727f0ee89554bff9ba51633c53d8c47)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:24.504059 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65e5b33-d458-45d2-8cfb-ae6695658087 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:24.561642 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1422/2856] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 04:00:24 2022] POST /identity/v3/users => generated 293 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 04:00:24.565867 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fec7789-47ab-4f65-adc6-b9199ced2ebb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:24.577567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fec7789-47ab-4f65-adc6-b9199ced2ebb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:24.582553 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bf296c-a0ee-4989-9e2b-0b8cc00c3c68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:24.583181 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bf296c-a0ee-4989-9e2b-0b8cc00c3c68 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:24.583458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bf296c-a0ee-4989-9e2b-0b8cc00c3c68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:24.583729 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bf296c-a0ee-4989-9e2b-0b8cc00c3c68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:24.584476 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bf296c-a0ee-4989-9e2b-0b8cc00c3c68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:24.586309 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bf296c-a0ee-4989-9e2b-0b8cc00c3c68 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:24.598595 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1423/2857] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 04:00:24 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 04:00:24.603877 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bf296c-a0ee-4989-9e2b-0b8cc00c3c68 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:24.614403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bf296c-a0ee-4989-9e2b-0b8cc00c3c68 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:24.619919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bcae447-4bbb-480f-a22d-7c08ba61e361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:24.620671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcae447-4bbb-480f-a22d-7c08ba61e361 admin admin] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:24.620946 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcae447-4bbb-480f-a22d-7c08ba61e361 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:24.621215 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcae447-4bbb-480f-a22d-7c08ba61e361 admin admin] PATH_INFO: `/v3/roles` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:24.621860 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bcae447-4bbb-480f-a22d-7c08ba61e361 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:24.623687 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bcae447-4bbb-480f-a22d-7c08ba61e361 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:24.635101 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1424/2858] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 04:00:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 24 04:00:24.639275 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bcae447-4bbb-480f-a22d-7c08ba61e361 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:24.649257 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bcae447-4bbb-480f-a22d-7c08ba61e361 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:24.654312 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84008df6-7251-4983-a342-067fae6a7505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:24.654989 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84008df6-7251-4983-a342-067fae6a7505 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:24.655268 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84008df6-7251-4983-a342-067fae6a7505 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:24.655555 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84008df6-7251-4983-a342-067fae6a7505 admin admin] PATH_INFO: `/v3/projects/e2c1183122314dbd8982c3d24a9b041e/users/541f59acaa84467783f63c278a7f0485/roles/9912acfb698e41ebbce6a5147fafe896` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:24.671067 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84008df6-7251-4983-a342-067fae6a7505 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2c1183122314dbd8982c3d24a9b041e, user_id=541f59acaa84467783f63c278a7f0485, role_id=9912acfb698e41ebbce6a5147fafe896)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:24.673691 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84008df6-7251-4983-a342-067fae6a7505 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:24.753383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1425/2859] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 04:00:24 2022] PUT /identity/v3/projects/e2c1183122314dbd8982c3d24a9b041e/users/541f59acaa84467783f63c278a7f0485/roles/9912acfb698e41ebbce6a5147fafe896 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 04:00:24.758755 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026edb2d-26ba-42e5-9672-8564f0ddb251 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:24.759041 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026edb2d-26ba-42e5-9672-8564f0ddb251 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:24.759315 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026edb2d-26ba-42e5-9672-8564f0ddb251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:24.778133 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-026edb2d-26ba-42e5-9672-8564f0ddb251 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:24.779526 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-026edb2d-26ba-42e5-9672-8564f0ddb251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:24.809580 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c65e5b33-d458-45d2-8cfb-ae6695658087 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a727f0ee89554bff9ba51633c53d8c47'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:24.810317 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c65e5b33-d458-45d2-8cfb-ae6695658087 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a727f0ee89554bff9ba51633c53d8c47'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:24.812523 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:24.817752 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c65e5b33-d458-45d2-8cfb-ae6695658087 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a727f0ee89554bff9ba51633c53d8c47'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:24.818422 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c65e5b33-d458-45d2-8cfb-ae6695658087 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a727f0ee89554bff9ba51633c53d8c47'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:24.842102 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1426/2860] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 04:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 04:00:24.852733 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026edb2d-26ba-42e5-9672-8564f0ddb251 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:24.854081 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-026edb2d-26ba-42e5-9672-8564f0ddb251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:24.867331 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:24.902117 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-c65e5b33-d458-45d2-8cfb-ae6695658087 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a727f0ee89554bff9ba51633c53d8c47'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:24.914423 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026edb2d-26ba-42e5-9672-8564f0ddb251 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:24.918484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19053641-a37d-4f33-86c4-fc25f15898a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:24.919087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19053641-a37d-4f33-86c4-fc25f15898a1 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:24.919361 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19053641-a37d-4f33-86c4-fc25f15898a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:24.919631 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19053641-a37d-4f33-86c4-fc25f15898a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:24.922973 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1435/2861] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 04:00:24 2022] DELETE /identity/v3/users/a727f0ee89554bff9ba51633c53d8c47 => generated 0 bytes in 442 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 04:00:24.930094 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fc2e54-1a83-4a0c-84fd-98321892c119 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:24.930426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fc2e54-1a83-4a0c-84fd-98321892c119 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:24.930702 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fc2e54-1a83-4a0c-84fd-98321892c119 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:24.931824 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19053641-a37d-4f33-86c4-fc25f15898a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:24.933296 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19053641-a37d-4f33-86c4-fc25f15898a1 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:24.944954 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1427/2862] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:00:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 04:00:24.950506 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-58fc2e54-1a83-4a0c-84fd-98321892c119 None None] MFA Rules not processed for user `c73de3f5e3b94a3182c45d2476903a00`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:24.951700 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-58fc2e54-1a83-4a0c-84fd-98321892c119 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:25.012865 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1436/2863] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 04:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 04:00:25.023087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19053641-a37d-4f33-86c4-fc25f15898a1 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:25.024612 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-19053641-a37d-4f33-86c4-fc25f15898a1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:25.035977 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19053641-a37d-4f33-86c4-fc25f15898a1 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:25.088701 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4d3711-a48d-4638-9484-246fba0fc8a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:25.089232 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4d3711-a48d-4638-9484-246fba0fc8a1 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:25.089511 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4d3711-a48d-4638-9484-246fba0fc8a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:25.089805 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4d3711-a48d-4638-9484-246fba0fc8a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:25.099229 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4d3711-a48d-4638-9484-246fba0fc8a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:25.101594 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4d3711-a48d-4638-9484-246fba0fc8a1 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:25.111504 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1428/2864] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:00:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:00:25.859397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fc2e54-1a83-4a0c-84fd-98321892c119 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:25.861585 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-58fc2e54-1a83-4a0c-84fd-98321892c119 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:25.877840 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fc2e54-1a83-4a0c-84fd-98321892c119 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:25.882964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:25.883766 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:25.884143 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:25.884521 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] PATH_INFO: `/v3/projects/640210fe7e55404c9bdc0dcbadc7252c` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:25.886605 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=640210fe7e55404c9bdc0dcbadc7252c)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:25.888846 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:25.996311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '640210fe7e55404c9bdc0dcbadc7252c', 'request_id': 'req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:26.001815 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '640210fe7e55404c9bdc0dcbadc7252c', 'request_id': 'req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:26.085541 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '640210fe7e55404c9bdc0dcbadc7252c', 'request_id': 'req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:26.097903 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] The token cache is being invalidate because project 640210fe7e55404c9bdc0dcbadc7252c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 04:00:26.098307 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:26.099139 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:00:26.102200 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1437/2865] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 04:00:25 2022] DELETE /identity/v3/projects/640210fe7e55404c9bdc0dcbadc7252c => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:00:30.445271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f415dc-a16e-4adf-ac5b-7a3bdc0dbc73 None None] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:30.445760 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f415dc-a16e-4adf-ac5b-7a3bdc0dbc73 None None] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:30.446007 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f415dc-a16e-4adf-ac5b-7a3bdc0dbc73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:30.476501 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.auth.core [None req-69f415dc-a16e-4adf-ac5b-7a3bdc0dbc73 None None] MFA Rules not processed for user `541f59acaa84467783f63c278a7f0485`. Rule list: `[]` (Enabled: `True`). {{(pid=81658) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:30.477809 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-69f415dc-a16e-4adf-ac5b-7a3bdc0dbc73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:30.544565 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1429/2866] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 04:00:30 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 04:00:30.549013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:30.550600 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:30.602827 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:30.615343 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86da5b8b-662b-484c-a58a-7f4d82b5e2ee admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:30.620432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f164adf4-14ee-4044-ba12-d7c60e5e6795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:30.626357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f164adf4-14ee-4044-ba12-d7c60e5e6795 admin admin] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:30.626357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f164adf4-14ee-4044-ba12-d7c60e5e6795 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:30.626357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f164adf4-14ee-4044-ba12-d7c60e5e6795 admin admin] PATH_INFO: `/v3/projects` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:30.626357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f164adf4-14ee-4044-ba12-d7c60e5e6795 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:30.626357 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f164adf4-14ee-4044-ba12-d7c60e5e6795 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:30.721087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1438/2867] 10.222.0.65 () {60 vars in 1153 bytes} [Wed Aug 24 04:00:30 2022] POST /identity/v3/projects => generated 383 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 24 04:00:30.725175 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f415dc-a16e-4adf-ac5b-7a3bdc0dbc73 None None] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:30.736399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f415dc-a16e-4adf-ac5b-7a3bdc0dbc73 None None] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:30.781054 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:30.793644 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824877c9-7c7e-45cb-8cdc-55863dd4c156 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:30.794283 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824877c9-7c7e-45cb-8cdc-55863dd4c156 admin admin] REQUEST_METHOD: `POST` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:30.794564 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824877c9-7c7e-45cb-8cdc-55863dd4c156 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:30.794855 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824877c9-7c7e-45cb-8cdc-55863dd4c156 admin admin] PATH_INFO: `/v3/users` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:30.795483 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824877c9-7c7e-45cb-8cdc-55863dd4c156 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:30.797389 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824877c9-7c7e-45cb-8cdc-55863dd4c156 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:30.868822 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1430/2868] 10.222.0.65 () {60 vars in 1144 bytes} [Wed Aug 24 04:00:30 2022] POST /identity/v3/users => generated 298 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 24 04:00:30.873690 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f164adf4-14ee-4044-ba12-d7c60e5e6795 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:30.885948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f164adf4-14ee-4044-ba12-d7c60e5e6795 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:30.891440 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ebb6e6-23c2-4595-aa0c-8fecf0394f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:30.892057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ebb6e6-23c2-4595-aa0c-8fecf0394f5f admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:30.892335 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ebb6e6-23c2-4595-aa0c-8fecf0394f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:30.892609 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ebb6e6-23c2-4595-aa0c-8fecf0394f5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:30.893274 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ebb6e6-23c2-4595-aa0c-8fecf0394f5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:30.896663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ebb6e6-23c2-4595-aa0c-8fecf0394f5f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:30.910047 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1439/2869] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 04:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 04:00:30.915087 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824877c9-7c7e-45cb-8cdc-55863dd4c156 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:30.939381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824877c9-7c7e-45cb-8cdc-55863dd4c156 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:30.945670 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfb46d2-7811-4382-9d4b-a75b0a657c26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:30.946449 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfb46d2-7811-4382-9d4b-a75b0a657c26 admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:30.946792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfb46d2-7811-4382-9d4b-a75b0a657c26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:30.947159 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfb46d2-7811-4382-9d4b-a75b0a657c26 admin admin] PATH_INFO: `/v3/projects/74abf8f83fa543439073d53e184d010c/users/7bf6e0523b2f4d64ba5d1f7232775b24/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:30.965515 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfb46d2-7811-4382-9d4b-a75b0a657c26 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74abf8f83fa543439073d53e184d010c, user_id=7bf6e0523b2f4d64ba5d1f7232775b24, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:30.968263 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfb46d2-7811-4382-9d4b-a75b0a657c26 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:31.044033 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1431/2870] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 04:00:30 2022] PUT /identity/v3/projects/74abf8f83fa543439073d53e184d010c/users/7bf6e0523b2f4d64ba5d1f7232775b24/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:00:31.048724 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ebb6e6-23c2-4595-aa0c-8fecf0394f5f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:31.059053 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ebb6e6-23c2-4595-aa0c-8fecf0394f5f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:31.124572 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081578d6-be43-4302-bb07-e028a3b7185e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:31.125635 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081578d6-be43-4302-bb07-e028a3b7185e admin admin] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:31.125924 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081578d6-be43-4302-bb07-e028a3b7185e admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:31.126230 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081578d6-be43-4302-bb07-e028a3b7185e admin admin] PATH_INFO: `/v3/roles` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:31.126950 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081578d6-be43-4302-bb07-e028a3b7185e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:31.128577 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081578d6-be43-4302-bb07-e028a3b7185e admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:31.140796 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1440/2871] 10.222.0.65 () {58 vars in 1122 bytes} [Wed Aug 24 04:00:31 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 24 04:00:31.145211 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfb46d2-7811-4382-9d4b-a75b0a657c26 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:31.158092 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfb46d2-7811-4382-9d4b-a75b0a657c26 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:31.169241 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecb994e-8f56-47b5-b619-bedf9d3c6a2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:31.170408 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecb994e-8f56-47b5-b619-bedf9d3c6a2a admin admin] REQUEST_METHOD: `PUT` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:31.170813 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecb994e-8f56-47b5-b619-bedf9d3c6a2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:31.171091 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecb994e-8f56-47b5-b619-bedf9d3c6a2a admin admin] PATH_INFO: `/v3/projects/74abf8f83fa543439073d53e184d010c/users/7bf6e0523b2f4d64ba5d1f7232775b24/roles/43ca25467bc6484da3a50508efb79230` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:31.173376 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecb994e-8f56-47b5-b619-bedf9d3c6a2a admin admin] RBAC: Authorizing `identity:create_grant(project_id=74abf8f83fa543439073d53e184d010c, user_id=7bf6e0523b2f4d64ba5d1f7232775b24, role_id=43ca25467bc6484da3a50508efb79230)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:31.176090 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecb994e-8f56-47b5-b619-bedf9d3c6a2a admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:31.188458 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1432/2872] 10.222.0.65 () {60 vars in 1483 bytes} [Wed Aug 24 04:00:31 2022] PUT /identity/v3/projects/74abf8f83fa543439073d53e184d010c/users/7bf6e0523b2f4d64ba5d1f7232775b24/roles/43ca25467bc6484da3a50508efb79230 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:00:36.742581 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3171c3f0-f0c5-4b46-b00a-4e3f2626dca3 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:36.743503 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3171c3f0-f0c5-4b46-b00a-4e3f2626dca3 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:36.743779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3171c3f0-f0c5-4b46-b00a-4e3f2626dca3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:36.774744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-3171c3f0-f0c5-4b46-b00a-4e3f2626dca3 None None] MFA Rules not processed for user `7bf6e0523b2f4d64ba5d1f7232775b24`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:36.775169 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-3171c3f0-f0c5-4b46-b00a-4e3f2626dca3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:36.843303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1441/2873] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 04:00:36 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 04:00:36.855839 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecb994e-8f56-47b5-b619-bedf9d3c6a2a admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:36.857634 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5ecb994e-8f56-47b5-b619-bedf9d3c6a2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:36.918628 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecb994e-8f56-47b5-b619-bedf9d3c6a2a admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:36.922306 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575b817c-07cf-4501-b9da-3a9e80cfeed7 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f3b3c7ebc08496e970f8f25fd6395e4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:36.925027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575b817c-07cf-4501-b9da-3a9e80cfeed7 service glance] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:36.925027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575b817c-07cf-4501-b9da-3a9e80cfeed7 service glance] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:36.925027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575b817c-07cf-4501-b9da-3a9e80cfeed7 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:36.987448 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575b817c-07cf-4501-b9da-3a9e80cfeed7 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:36.988985 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575b817c-07cf-4501-b9da-3a9e80cfeed7 service glance] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:36.999472 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1433/2874] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:00:36 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:00:37.281201 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dc4668-f42c-4f0d-a978-c0c9750195a1 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:37.281485 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dc4668-f42c-4f0d-a978-c0c9750195a1 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:37.281760 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dc4668-f42c-4f0d-a978-c0c9750195a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:37.338250 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-72dc4668-f42c-4f0d-a978-c0c9750195a1 None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:37.339515 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-72dc4668-f42c-4f0d-a978-c0c9750195a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:37.358124 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:37.419679 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1442/2875] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 04:00:37 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 04:00:37.546480 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575b817c-07cf-4501-b9da-3a9e80cfeed7 service glance] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:37.548027 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-575b817c-07cf-4501-b9da-3a9e80cfeed7 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:37.616383 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:37.652049 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575b817c-07cf-4501-b9da-3a9e80cfeed7 service glance] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:37.658566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8696c65-2b30-4570-8464-6e0f319daa26 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:37.659127 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8696c65-2b30-4570-8464-6e0f319daa26 service swift] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:37.659404 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8696c65-2b30-4570-8464-6e0f319daa26 service swift] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:37.660252 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8696c65-2b30-4570-8464-6e0f319daa26 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:37.669629 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8696c65-2b30-4570-8464-6e0f319daa26 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:37.671274 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8696c65-2b30-4570-8464-6e0f319daa26 service swift] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:37.680439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1434/2876] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 04:00:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 04:00:47.419919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622132ea-d567-4a5e-92a3-9624e66d29b5 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:47.419919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622132ea-d567-4a5e-92a3-9624e66d29b5 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:47.419919 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622132ea-d567-4a5e-92a3-9624e66d29b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:47.438995 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-622132ea-d567-4a5e-92a3-9624e66d29b5 None None] MFA Rules not processed for user `541f59acaa84467783f63c278a7f0485`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:47.440207 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-622132ea-d567-4a5e-92a3-9624e66d29b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:47.447947 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1443/2877] 10.222.0.65 () {58 vars in 958 bytes} [Wed Aug 24 04:00:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 24 04:00:47.676080 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=17 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:00:47.732937 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8696c65-2b30-4570-8464-6e0f319daa26 service swift] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:47.734435 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a8696c65-2b30-4570-8464-6e0f319daa26 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:47.792374 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8696c65-2b30-4570-8464-6e0f319daa26 service swift] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:47.796006 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d1a507-52bb-4223-97db-417f31585f67 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:47.796649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d1a507-52bb-4223-97db-417f31585f67 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:47.796933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d1a507-52bb-4223-97db-417f31585f67 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:47.797203 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d1a507-52bb-4223-97db-417f31585f67 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:47.798988 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-88d1a507-52bb-4223-97db-417f31585f67 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:47.808671 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d1a507-52bb-4223-97db-417f31585f67 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:47.810146 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d1a507-52bb-4223-97db-417f31585f67 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:47.818403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1435/2878] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:00:47 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:00:51.047127 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0c5d19-a39a-46fb-954e-a16c249159e8 None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:51.047611 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0c5d19-a39a-46fb-954e-a16c249159e8 None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:51.047722 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0c5d19-a39a-46fb-954e-a16c249159e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:51.066682 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-6b0c5d19-a39a-46fb-954e-a16c249159e8 None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:00:51.067897 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6b0c5d19-a39a-46fb-954e-a16c249159e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:00:51.076057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1444/2879] 10.222.0.65 () {58 vars in 1009 bytes} [Wed Aug 24 04:00:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 04:00:51.088205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d1a507-52bb-4223-97db-417f31585f67 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:00:51.101258 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d1a507-52bb-4223-97db-417f31585f67 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:00:51.106477 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e1c0b7-08c6-4c72-b3a0-9616d456075c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:00:51.107079 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e1c0b7-08c6-4c72-b3a0-9616d456075c service swift] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:00:51.107354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e1c0b7-08c6-4c72-b3a0-9616d456075c service swift] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:00:51.107621 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e1c0b7-08c6-4c72-b3a0-9616d456075c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:00:51.116000 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e1c0b7-08c6-4c72-b3a0-9616d456075c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:00:51.118373 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e1c0b7-08c6-4c72-b3a0-9616d456075c service swift] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:00:51.126694 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1436/2880] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 04:00:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 04:01:01.061427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=25 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:01:01.123400 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=32 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:01:09.643041 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0c5d19-a39a-46fb-954e-a16c249159e8 None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:01:09.654075 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6b0c5d19-a39a-46fb-954e-a16c249159e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:01:09.719439 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0c5d19-a39a-46fb-954e-a16c249159e8 None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:01:09.724940 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1081f324-b746-4841-9f55-912b705c63ee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:01:09.725797 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1081f324-b746-4841-9f55-912b705c63ee service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:01:09.726115 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1081f324-b746-4841-9f55-912b705c63ee service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:01:09.726414 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1081f324-b746-4841-9f55-912b705c63ee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:01:09.728142 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1081f324-b746-4841-9f55-912b705c63ee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:01:09.742178 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1081f324-b746-4841-9f55-912b705c63ee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:01:09.743798 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1081f324-b746-4841-9f55-912b705c63ee service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:01:09.755036 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1445/2881] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:01:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:01:19.750329 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=34 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:01:27.235172 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e1c0b7-08c6-4c72-b3a0-9616d456075c service swift] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:01:27.245056 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e1c0b7-08c6-4c72-b3a0-9616d456075c service swift] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:01:27.249640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d615a1-eebf-46c4-bc3d-14cfe1aeaba0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:01:27.250281 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d615a1-eebf-46c4-bc3d-14cfe1aeaba0 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:01:27.250566 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d615a1-eebf-46c4-bc3d-14cfe1aeaba0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:01:27.250842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d615a1-eebf-46c4-bc3d-14cfe1aeaba0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:01:27.260165 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d615a1-eebf-46c4-bc3d-14cfe1aeaba0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:01:27.261742 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d615a1-eebf-46c4-bc3d-14cfe1aeaba0 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:01:27.271591 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1437/2882] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:01:27 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:01:32.179066 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1081f324-b746-4841-9f55-912b705c63ee service nova] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:01:32.179066 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-1081f324-b746-4841-9f55-912b705c63ee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:01:32.196321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1081f324-b746-4841-9f55-912b705c63ee service nova] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:01:32.204366 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7d318e-a2fd-4c18-a1ce-7be31058c923 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5896018712d4b82a781365e478ee855', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:01:32.205745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7d318e-a2fd-4c18-a1ce-7be31058c923 service nova] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:01:32.206397 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7d318e-a2fd-4c18-a1ce-7be31058c923 service nova] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:01:32.210647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7d318e-a2fd-4c18-a1ce-7be31058c923 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:01:32.210647 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-cb7d318e-a2fd-4c18-a1ce-7be31058c923 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:01:32.224857 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7d318e-a2fd-4c18-a1ce-7be31058c923 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:01:32.227808 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7d318e-a2fd-4c18-a1ce-7be31058c923 service nova] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:01:32.249309 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1446/2883] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:01:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:01:37.267413 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=6 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:01:38.258034 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d615a1-eebf-46c4-bc3d-14cfe1aeaba0 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:01:38.259551 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-05d615a1-eebf-46c4-bc3d-14cfe1aeaba0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:01:38.271114 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d615a1-eebf-46c4-bc3d-14cfe1aeaba0 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:01:38.274740 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243c287b-18c9-47f6-84e9-21d4a98b5d3a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:01:38.275354 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243c287b-18c9-47f6-84e9-21d4a98b5d3a service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:01:38.275626 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243c287b-18c9-47f6-84e9-21d4a98b5d3a service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:01:38.275890 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243c287b-18c9-47f6-84e9-21d4a98b5d3a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:01:38.277582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-243c287b-18c9-47f6-84e9-21d4a98b5d3a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:01:38.340853 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243c287b-18c9-47f6-84e9-21d4a98b5d3a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:01:38.342967 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243c287b-18c9-47f6-84e9-21d4a98b5d3a service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:01:38.352929 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1438/2884] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:01:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 04:01:42.239138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=6 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:01:45.554433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7d318e-a2fd-4c18-a1ce-7be31058c923 service nova] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:01:45.555909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-cb7d318e-a2fd-4c18-a1ce-7be31058c923 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:01:45.629794 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7d318e-a2fd-4c18-a1ce-7be31058c923 service nova] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:01:45.635288 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b3999d-d18e-4954-a016-7dc28a65ed02 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:01:45.635948 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b3999d-d18e-4954-a016-7dc28a65ed02 service placement] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:01:45.636227 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b3999d-d18e-4954-a016-7dc28a65ed02 service placement] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:01:45.636499 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b3999d-d18e-4954-a016-7dc28a65ed02 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:01:45.638480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-50b3999d-d18e-4954-a016-7dc28a65ed02 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:01:45.650424 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b3999d-d18e-4954-a016-7dc28a65ed02 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:01:45.652057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b3999d-d18e-4954-a016-7dc28a65ed02 service placement] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:01:45.662765 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1447/2885] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:01:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:01:45.757923 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243c287b-18c9-47f6-84e9-21d4a98b5d3a service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:01:45.759484 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-243c287b-18c9-47f6-84e9-21d4a98b5d3a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:01:45.774285 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243c287b-18c9-47f6-84e9-21d4a98b5d3a service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:01:45.778976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:01:45.780429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:01:45.780880 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:01:45.781246 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] PATH_INFO: `/v3/users/71ff73cade8b42c7ae57be1e3ef0e028` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:01:45.796466 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] RBAC: Authorizing `identity:delete_user(user_id=71ff73cade8b42c7ae57be1e3ef0e028)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:01:45.798493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:01:45.960167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71ff73cade8b42c7ae57be1e3ef0e028'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:45.960801 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71ff73cade8b42c7ae57be1e3ef0e028'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:45.967897 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71ff73cade8b42c7ae57be1e3ef0e028'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:45.979878 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71ff73cade8b42c7ae57be1e3ef0e028'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:46.049810 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71ff73cade8b42c7ae57be1e3ef0e028'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:46.056180 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1439/2886] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 04:01:45 2022] DELETE /identity/v3/users/71ff73cade8b42c7ae57be1e3ef0e028 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:01:46.280433 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b3999d-d18e-4954-a016-7dc28a65ed02 service placement] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:01:46.291900 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b3999d-d18e-4954-a016-7dc28a65ed02 service placement] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:01:46.349589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:01:46.350586 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:01:46.350868 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:01:46.351140 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] PATH_INFO: `/v3/projects/33235ed2b4cd48e7ac205143b814c6fb` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:01:46.362586 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=33235ed2b4cd48e7ac205143b814c6fb)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:01:46.364878 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:01:46.486456 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33235ed2b4cd48e7ac205143b814c6fb', 'request_id': 'req-8c2f95ae-c149-4ee0-9801-68c83a4178bb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:46.492964 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33235ed2b4cd48e7ac205143b814c6fb', 'request_id': 'req-8c2f95ae-c149-4ee0-9801-68c83a4178bb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:46.558473 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33235ed2b4cd48e7ac205143b814c6fb', 'request_id': 'req-8c2f95ae-c149-4ee0-9801-68c83a4178bb'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:46.570488 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] The token cache is being invalidate because project 33235ed2b4cd48e7ac205143b814c6fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 04:01:46.570909 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:46.571550 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:46.573089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1448/2887] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 04:01:46 2022] DELETE /identity/v3/projects/33235ed2b4cd48e7ac205143b814c6fb => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:01:46.577399 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:01:46.578869 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:01:46.593095 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5479bd26-32fd-44aa-ba90-6a6f215be88e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:01:46.648919 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:01:46.656475 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:01:46.664758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:01:46.664758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:01:46.664758 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] PATH_INFO: `/v3/users/711466756ea645b7846b473fdc76e490` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:01:46.670619 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] RBAC: Authorizing `identity:delete_user(user_id=711466756ea645b7846b473fdc76e490)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:01:46.672540 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:01:46.858542 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '711466756ea645b7846b473fdc76e490'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:46.859216 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '711466756ea645b7846b473fdc76e490'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:46.866173 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '711466756ea645b7846b473fdc76e490'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:46.878422 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '711466756ea645b7846b473fdc76e490'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:46.947205 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '711466756ea645b7846b473fdc76e490'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:46.953550 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1440/2888] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 04:01:46 2022] DELETE /identity/v3/users/711466756ea645b7846b473fdc76e490 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 04:01:47.200189 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:01:47.202509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:01:47.217226 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2f95ae-c149-4ee0-9801-68c83a4178bb admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:01:47.226550 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:01:47.276013 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:01:47.276790 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:01:47.277070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:01:47.277342 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] PATH_INFO: `/v3/projects/4af87005146c4e9b9af06274a0235eaa` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:01:47.289057 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4af87005146c4e9b9af06274a0235eaa)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:01:47.291303 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:01:47.440496 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4af87005146c4e9b9af06274a0235eaa', 'request_id': 'req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:47.446015 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4af87005146c4e9b9af06274a0235eaa', 'request_id': 'req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:47.512698 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4af87005146c4e9b9af06274a0235eaa', 'request_id': 'req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:47.525138 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] The token cache is being invalidate because project 4af87005146c4e9b9af06274a0235eaa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 04:01:47.525426 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:47.526063 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:01:47.527660 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1449/2889] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 04:01:47 2022] DELETE /identity/v3/projects/4af87005146c4e9b9af06274a0235eaa => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:01:56.949730 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=33,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:01:57.522096 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=50,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:02:20.430014 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:02:20.431664 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:02:20.503267 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d07fc6-820b-4df8-b236-dd24bf70570e admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:02:20.563200 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d0c40c-77a7-439d-9b97-28d2d6e5100c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:02:20.563808 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d0c40c-77a7-439d-9b97-28d2d6e5100c service placement] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:02:20.564644 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d0c40c-77a7-439d-9b97-28d2d6e5100c service placement] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:02:20.564933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d0c40c-77a7-439d-9b97-28d2d6e5100c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:02:20.566739 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d0d0c40c-77a7-439d-9b97-28d2d6e5100c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:02:20.578280 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d0c40c-77a7-439d-9b97-28d2d6e5100c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:02:20.579907 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d0c40c-77a7-439d-9b97-28d2d6e5100c service placement] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:02:20.590197 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1441/2890] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:02:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:02:25.313221 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:02:25.314841 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:02:25.374691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629b95c1-64d3-454e-a5a0-ebc668a1ebe4 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:02:25.378848 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec3a79-180b-4517-a2a8-7570e523c023 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:02:25.379617 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec3a79-180b-4517-a2a8-7570e523c023 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:02:25.379957 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec3a79-180b-4517-a2a8-7570e523c023 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:02:25.380258 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec3a79-180b-4517-a2a8-7570e523c023 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:02:25.382070 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-d4ec3a79-180b-4517-a2a8-7570e523c023 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:02:25.391691 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ec3a79-180b-4517-a2a8-7570e523c023 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:02:25.393108 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ec3a79-180b-4517-a2a8-7570e523c023 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:02:25.401582 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1450/2891] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:02:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:02:25.486578 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d0c40c-77a7-439d-9b97-28d2d6e5100c service placement] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:02:25.496826 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d0c40c-77a7-439d-9b97-28d2d6e5100c service placement] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:02:25.500414 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddc3e53-7d2c-48c1-8e15-8f8aa8994581 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:02:25.500997 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddc3e53-7d2c-48c1-8e15-8f8aa8994581 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:02:25.501269 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddc3e53-7d2c-48c1-8e15-8f8aa8994581 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:02:25.501546 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddc3e53-7d2c-48c1-8e15-8f8aa8994581 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:02:25.504279 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-eddc3e53-7d2c-48c1-8e15-8f8aa8994581 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:02:25.574940 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddc3e53-7d2c-48c1-8e15-8f8aa8994581 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:02:25.576403 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddc3e53-7d2c-48c1-8e15-8f8aa8994581 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:02:25.587791 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1442/2892] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:02:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:02:35.398087 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=26 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:02:35.582981 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:02:58.371614 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef8fefa-f230-42e2-99a5-e79bb2580e0a None None] REQUEST_METHOD: `POST` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:02:58.372083 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef8fefa-f230-42e2-99a5-e79bb2580e0a None None] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:02:58.372193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef8fefa-f230-42e2-99a5-e79bb2580e0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:02:58.392720 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.auth.core [None req-cef8fefa-f230-42e2-99a5-e79bb2580e0a None None] MFA Rules not processed for user `c63ade06fed946e79dc1e4b5ecfab5e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81657) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 24 04:02:58.393904 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-cef8fefa-f230-42e2-99a5-e79bb2580e0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:02:58.452692 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1451/2893] 10.222.0.65 () {58 vars in 967 bytes} [Wed Aug 24 04:02:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 24 04:02:58.493096 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddc3e53-7d2c-48c1-8e15-8f8aa8994581 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:02:58.495219 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-eddc3e53-7d2c-48c1-8e15-8f8aa8994581 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:02:58.565404 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddc3e53-7d2c-48c1-8e15-8f8aa8994581 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:02:58.570521 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ab8b1c-bc07-4e8a-9bfa-d4fe33e63e4a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4591bec74a4641af818b2d75b657115e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:02:58.571113 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ab8b1c-bc07-4e8a-9bfa-d4fe33e63e4a service swift] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:02:58.571386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ab8b1c-bc07-4e8a-9bfa-d4fe33e63e4a service swift] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:02:58.571649 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ab8b1c-bc07-4e8a-9bfa-d4fe33e63e4a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:02:58.580183 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ab8b1c-bc07-4e8a-9bfa-d4fe33e63e4a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:02:58.581735 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ab8b1c-bc07-4e8a-9bfa-d4fe33e63e4a service swift] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:02:58.589831 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1443/2894] 10.222.0.65 () {60 vars in 1384 bytes} [Wed Aug 24 04:02:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 24 04:03:03.340443 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef8fefa-f230-42e2-99a5-e79bb2580e0a None None] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:03:03.341911 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-cef8fefa-f230-42e2-99a5-e79bb2580e0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:03:03.362105 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef8fefa-f230-42e2-99a5-e79bb2580e0a None None] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:03:03.362845 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28ff797-ee21-4c2c-8391-3008347e8448 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:03:03.363466 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28ff797-ee21-4c2c-8391-3008347e8448 service placement] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:03:03.363745 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28ff797-ee21-4c2c-8391-3008347e8448 service placement] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:03:03.364031 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28ff797-ee21-4c2c-8391-3008347e8448 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:03:03.367460 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a28ff797-ee21-4c2c-8391-3008347e8448 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:03:03.379709 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28ff797-ee21-4c2c-8391-3008347e8448 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:03:03.381284 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28ff797-ee21-4c2c-8391-3008347e8448 service placement] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:03:03.392285 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1452/2895] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:03:03 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:03:06.848493 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ab8b1c-bc07-4e8a-9bfa-d4fe33e63e4a service swift] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:03:06.850211 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-80ab8b1c-bc07-4e8a-9bfa-d4fe33e63e4a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:03:06.918815 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ab8b1c-bc07-4e8a-9bfa-d4fe33e63e4a service swift] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:03:06.923337 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:03:06.923960 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:03:06.924239 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:03:06.924511 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] PATH_INFO: `/v3/users/541f59acaa84467783f63c278a7f0485` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:03:06.925627 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] RBAC: Authorizing `identity:delete_user(user_id=541f59acaa84467783f63c278a7f0485)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:03:06.927575 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:03:07.154791 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '541f59acaa84467783f63c278a7f0485'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:07.155439 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '541f59acaa84467783f63c278a7f0485'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:07.162394 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '541f59acaa84467783f63c278a7f0485'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:07.174654 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '541f59acaa84467783f63c278a7f0485'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:07.261405 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '541f59acaa84467783f63c278a7f0485'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:07.267906 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1444/2896] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 04:03:06 2022] DELETE /identity/v3/users/541f59acaa84467783f63c278a7f0485 => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:03:07.588398 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28ff797-ee21-4c2c-8391-3008347e8448 service placement] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:03:07.601118 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28ff797-ee21-4c2c-8391-3008347e8448 service placement] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:03:07.661810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:03:07.662460 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:03:07.662734 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:03:07.663026 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] PATH_INFO: `/v3/projects/e2c1183122314dbd8982c3d24a9b041e` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:03:07.664101 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2c1183122314dbd8982c3d24a9b041e)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:03:07.666095 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:03:07.770167 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2c1183122314dbd8982c3d24a9b041e', 'request_id': 'req-136f0543-c258-41bc-a1ef-64e81806b5ab'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:07.776836 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2c1183122314dbd8982c3d24a9b041e', 'request_id': 'req-136f0543-c258-41bc-a1ef-64e81806b5ab'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:07.848244 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2c1183122314dbd8982c3d24a9b041e', 'request_id': 'req-136f0543-c258-41bc-a1ef-64e81806b5ab'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:07.859744 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] The token cache is being invalidate because project e2c1183122314dbd8982c3d24a9b041e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 04:03:07.860032 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:07.860660 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:07.862122 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1453/2897] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 04:03:07 2022] DELETE /identity/v3/projects/e2c1183122314dbd8982c3d24a9b041e => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:03:07.866271 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:03:07.867615 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:03:07.880592 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12db9865-d444-4729-989b-4136a479fe08 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:03:07.885802 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:03:07.938214 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:03:07.938842 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:03:07.939116 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:03:07.939381 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] PATH_INFO: `/v3/users/7bf6e0523b2f4d64ba5d1f7232775b24` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:03:07.940499 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7bf6e0523b2f4d64ba5d1f7232775b24)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:03:07.942429 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:03:08.177567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7bf6e0523b2f4d64ba5d1f7232775b24'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:08.178188 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7bf6e0523b2f4d64ba5d1f7232775b24'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:08.185939 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7bf6e0523b2f4d64ba5d1f7232775b24'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:08.199151 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7bf6e0523b2f4d64ba5d1f7232775b24'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:08.307539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7bf6e0523b2f4d64ba5d1f7232775b24'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:08.314358 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1445/2898] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 04:03:07 2022] DELETE /identity/v3/users/7bf6e0523b2f4d64ba5d1f7232775b24 => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 24 04:03:08.593480 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:03:08.594967 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:03:08.609163 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136f0543-c258-41bc-a1ef-64e81806b5ab admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:03:08.629349 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:03:08.668565 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:03:08.669225 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:03:08.669507 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:03:08.669779 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] PATH_INFO: `/v3/projects/74abf8f83fa543439073d53e184d010c` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:03:08.670963 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=74abf8f83fa543439073d53e184d010c)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:03:08.672953 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:03:08.781402 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74abf8f83fa543439073d53e184d010c', 'request_id': 'req-5248200c-cc5d-4f8b-8852-60a1552da4a7'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:08.786837 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74abf8f83fa543439073d53e184d010c', 'request_id': 'req-5248200c-cc5d-4f8b-8852-60a1552da4a7'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:08.843761 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74abf8f83fa543439073d53e184d010c', 'request_id': 'req-5248200c-cc5d-4f8b-8852-60a1552da4a7'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:08.857331 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] The token cache is being invalidate because project 74abf8f83fa543439073d53e184d010c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81657) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 04:03:08.857619 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:08.858321 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:03:08.859914 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1454/2899] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 04:03:08 2022] DELETE /identity/v3/projects/74abf8f83fa543439073d53e184d010c => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:03:18.310201 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=49,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:03:18.941509 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=41,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:04:04.401471 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:04:04.402966 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:04:04.460539 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0299c7e6-457a-479c-b472-150a75914fa8 admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:04:04.508530 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881bcdab-ea70-4972-ad84-9eee63acbed3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:04:04.509129 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881bcdab-ea70-4972-ad84-9eee63acbed3 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:04:04.509404 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881bcdab-ea70-4972-ad84-9eee63acbed3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:04:04.509676 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881bcdab-ea70-4972-ad84-9eee63acbed3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:04:04.511416 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-881bcdab-ea70-4972-ad84-9eee63acbed3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:04:04.569864 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881bcdab-ea70-4972-ad84-9eee63acbed3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:04:04.571386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881bcdab-ea70-4972-ad84-9eee63acbed3 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:04:04.581060 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1446/2900] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:04:04 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:04:05.808279 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:04:05.813436 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:04:05.823168 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5248200c-cc5d-4f8b-8852-60a1552da4a7 admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:04:05.827478 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33e727e-35ff-4392-9771-a317c4c63f98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:04:05.828083 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33e727e-35ff-4392-9771-a317c4c63f98 service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:04:05.828360 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33e727e-35ff-4392-9771-a317c4c63f98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:04:05.828628 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33e727e-35ff-4392-9771-a317c4c63f98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:04:05.830427 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-a33e727e-35ff-4392-9771-a317c4c63f98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:04:05.840665 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33e727e-35ff-4392-9771-a317c4c63f98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:04:05.842968 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33e727e-35ff-4392-9771-a317c4c63f98 service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:04:05.851654 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1455/2901] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:04:05 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:04:06.112452 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881bcdab-ea70-4972-ad84-9eee63acbed3 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:04:06.113894 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-881bcdab-ea70-4972-ad84-9eee63acbed3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:04:06.125282 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881bcdab-ea70-4972-ad84-9eee63acbed3 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:04:06.129265 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a4f6b3-fefa-413a-94fa-5540061dc318 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:04:06.132933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a4f6b3-fefa-413a-94fa-5540061dc318 service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:04:06.132933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a4f6b3-fefa-413a-94fa-5540061dc318 service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:04:06.132933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a4f6b3-fefa-413a-94fa-5540061dc318 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:04:06.132933 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-99a4f6b3-fefa-413a-94fa-5540061dc318 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:04:06.204130 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=100 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:04:06.206498 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a4f6b3-fefa-413a-94fa-5540061dc318 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:04:06.207941 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a4f6b3-fefa-413a-94fa-5540061dc318 service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:04:06.219866 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1447/2902] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:04:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:04:07.403933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33e727e-35ff-4392-9771-a317c4c63f98 service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:04:07.416134 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33e727e-35ff-4392-9771-a317c4c63f98 service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:04:07.420502 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c48375b-e0cd-4407-ae4e-83628238de9e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:04:07.421193 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c48375b-e0cd-4407-ae4e-83628238de9e service neutron] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:04:07.421477 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c48375b-e0cd-4407-ae4e-83628238de9e service neutron] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:04:07.421747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c48375b-e0cd-4407-ae4e-83628238de9e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:04:07.423543 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8c48375b-e0cd-4407-ae4e-83628238de9e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:04:07.438824 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c48375b-e0cd-4407-ae4e-83628238de9e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:04:07.440260 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c48375b-e0cd-4407-ae4e-83628238de9e service neutron] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:04:07.451725 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1456/2903] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:04:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 24 04:04:11.165748 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a4f6b3-fefa-413a-94fa-5540061dc318 service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:04:11.174601 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a4f6b3-fefa-413a-94fa-5540061dc318 service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:04:11.178201 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cf18fe-4dfb-47b3-8afd-dbe05f2d009d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '64f23f0c8abc406b9700361723623387', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:04:11.178793 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cf18fe-4dfb-47b3-8afd-dbe05f2d009d service neutron] REQUEST_METHOD: `GET` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:04:11.179069 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cf18fe-4dfb-47b3-8afd-dbe05f2d009d service neutron] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:04:11.179804 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cf18fe-4dfb-47b3-8afd-dbe05f2d009d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:04:11.181666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-a4cf18fe-4dfb-47b3-8afd-dbe05f2d009d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:04:11.245196 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cf18fe-4dfb-47b3-8afd-dbe05f2d009d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:04:11.246675 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cf18fe-4dfb-47b3-8afd-dbe05f2d009d service neutron] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:04:11.256406 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1448/2904] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:04:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:04:17.446559 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=12 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:04:18.358232 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c48375b-e0cd-4407-ae4e-83628238de9e service neutron] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:04:18.359706 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-8c48375b-e0cd-4407-ae4e-83628238de9e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:04:18.374705 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c48375b-e0cd-4407-ae4e-83628238de9e service neutron] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:04:18.379381 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:04:18.380009 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:04:18.380298 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:04:18.380558 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] PATH_INFO: `/v3/users/6683479f4e1c419e950fa7497f194ac1` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:04:18.381676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] RBAC: Authorizing `identity:delete_user(user_id=6683479f4e1c419e950fa7497f194ac1)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:04:18.383625 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:04:18.548663 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6683479f4e1c419e950fa7497f194ac1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:18.549330 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6683479f4e1c419e950fa7497f194ac1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:18.556234 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6683479f4e1c419e950fa7497f194ac1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:18.556895 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6683479f4e1c419e950fa7497f194ac1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:18.620589 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6683479f4e1c419e950fa7497f194ac1'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:18.640133 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1457/2905] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 04:04:18 2022] DELETE /identity/v3/users/6683479f4e1c419e950fa7497f194ac1 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:04:18.891083 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cf18fe-4dfb-47b3-8afd-dbe05f2d009d service neutron] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:04:18.901432 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cf18fe-4dfb-47b3-8afd-dbe05f2d009d service neutron] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:04:18.960531 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:04:18.961085 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:04:18.961366 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:04:18.961640 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] PATH_INFO: `/v3/projects/7a6382d0a8d141e9ab64a41ae43b12af` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:04:18.962766 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a6382d0a8d141e9ab64a41ae43b12af)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:04:18.964666 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:04:19.099518 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a6382d0a8d141e9ab64a41ae43b12af', 'request_id': 'req-d53a3573-fa42-415b-bd3a-88183e1fb3bb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:19.105460 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a6382d0a8d141e9ab64a41ae43b12af', 'request_id': 'req-d53a3573-fa42-415b-bd3a-88183e1fb3bb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:19.113168 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a6382d0a8d141e9ab64a41ae43b12af', 'request_id': 'req-d53a3573-fa42-415b-bd3a-88183e1fb3bb'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:19.231585 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] The token cache is being invalidate because project 7a6382d0a8d141e9ab64a41ae43b12af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 04:04:19.231798 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:19.232752 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:19.234307 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1449/2906] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 04:04:18 2022] DELETE /identity/v3/projects/7a6382d0a8d141e9ab64a41ae43b12af => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:04:19.238388 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:04:19.239818 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:04:19.253146 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4dfbe93-969c-4bac-9c4d-5c2ccbe1d65f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:04:19.309758 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:04:19.310670 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:04:19.310949 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:04:19.311219 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] PATH_INFO: `/v3/users/cac8705e5db7411d8342e69c28f1fbe2` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:04:19.312361 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] RBAC: Authorizing `identity:delete_user(user_id=cac8705e5db7411d8342e69c28f1fbe2)` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:04:19.314311 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:04:19.441001 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cac8705e5db7411d8342e69c28f1fbe2'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:19.441667 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cac8705e5db7411d8342e69c28f1fbe2'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:19.449088 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cac8705e5db7411d8342e69c28f1fbe2'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:19.449747 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cac8705e5db7411d8342e69c28f1fbe2'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:19.506453 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.notifications [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cac8705e5db7411d8342e69c28f1fbe2'} {{(pid=81657) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:19.515676 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:04:19.525295 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1458/2907] 10.222.0.65 () {58 vars in 1224 bytes} [Wed Aug 24 04:04:19 2022] DELETE /identity/v3/users/cac8705e5db7411d8342e69c28f1fbe2 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:04:19.770582 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] Authenticating user token {{(pid=81658) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:04:19.772230 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.fernet_utils [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81658) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:04:19.786138 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53a3573-fa42-415b-bd3a-88183e1fb3bb admin admin] Validating token access rules against request {{(pid=81658) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:04:19.804382 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:04:19.846213 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ed0dc0-5948-48db-8d0e-87343fd8d632 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c73de3f5e3b94a3182c45d2476903a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8d28080f87c4783b9ef971b31a0d51a', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81658) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:04:19.846875 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ed0dc0-5948-48db-8d0e-87343fd8d632 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:04:19.847154 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ed0dc0-5948-48db-8d0e-87343fd8d632 admin admin] SCRIPT_NAME: `/identity` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:04:19.847425 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ed0dc0-5948-48db-8d0e-87343fd8d632 admin admin] PATH_INFO: `/v3/projects/c98e33b751bf493781505f6c0e47cf33` {{(pid=81658) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:04:19.848805 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ed0dc0-5948-48db-8d0e-87343fd8d632 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c98e33b751bf493781505f6c0e47cf33)` {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:04:19.850787 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ed0dc0-5948-48db-8d0e-87343fd8d632 admin admin] RBAC: Authorization granted {{(pid=81658) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:04:19.952792 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-80ed0dc0-5948-48db-8d0e-87343fd8d632 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c98e33b751bf493781505f6c0e47cf33', 'request_id': 'req-80ed0dc0-5948-48db-8d0e-87343fd8d632'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:19.958303 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-80ed0dc0-5948-48db-8d0e-87343fd8d632 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c98e33b751bf493781505f6c0e47cf33', 'request_id': 'req-80ed0dc0-5948-48db-8d0e-87343fd8d632'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:19.964567 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-80ed0dc0-5948-48db-8d0e-87343fd8d632 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c98e33b751bf493781505f6c0e47cf33', 'request_id': 'req-80ed0dc0-5948-48db-8d0e-87343fd8d632'} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:20.049167 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-80ed0dc0-5948-48db-8d0e-87343fd8d632 admin admin] The token cache is being invalidate because project c98e33b751bf493781505f6c0e47cf33 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81658) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 24 04:04:20.049386 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-80ed0dc0-5948-48db-8d0e-87343fd8d632 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:20.049976 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG keystone.notifications [None req-80ed0dc0-5948-48db-8d0e-87343fd8d632 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81658) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 24 04:04:20.051613 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: [pid: 81658|app: 0|req: 1450/2908] 10.222.0.65 () {58 vars in 1233 bytes} [Wed Aug 24 04:04:19 2022] DELETE /identity/v3/projects/c98e33b751bf493781505f6c0e47cf33 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 24 04:04:29.614908 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=3,keystone:DELETE=1 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:04:30.044522 dv-cinder-854259-1-iscsi devstack@keystone.service[81658]: DEBUG dbcounter [-] [81658] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81658) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 24 04:07:28.277406 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] Authenticating user token {{(pid=81657) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 24 04:07:28.278933 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:07:28.420713 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2bd639-3e51-46ac-a6a2-dd650c5a802f admin admin] Validating token access rules against request {{(pid=81657) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 24 04:07:28.480208 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6927035b-d670-40d1-8108-31a00885087a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b347c9c89a64168b51c439b4a92d647', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '5c653d5fa4304f298a295ce635efd5e6', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81657) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 24 04:07:28.480810 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6927035b-d670-40d1-8108-31a00885087a service placement] REQUEST_METHOD: `GET` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 24 04:07:28.481084 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6927035b-d670-40d1-8108-31a00885087a service placement] SCRIPT_NAME: `/identity` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 24 04:07:28.481355 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6927035b-d670-40d1-8108-31a00885087a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81657) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 24 04:07:28.483089 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.fernet_utils [None req-6927035b-d670-40d1-8108-31a00885087a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81657) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 24 04:07:28.494432 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6927035b-d670-40d1-8108-31a00885087a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 24 04:07:28.495975 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6927035b-d670-40d1-8108-31a00885087a service placement] RBAC: Authorization granted {{(pid=81657) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 24 04:07:28.506481 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: [pid: 81657|app: 0|req: 1459/2909] 10.222.0.65 () {60 vars in 1365 bytes} [Wed Aug 24 04:07:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 24 04:07:38.502373 dv-cinder-854259-1-iscsi devstack@keystone.service[81657]: DEBUG dbcounter [-] [81657] Writing DB stats keystone:SELECT=89 {{(pid=81657) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}